text:
tag:blogger.com,1999:blog-19369425923433970992014-08-22T23:58:15.497-07:00DataNucleusFlexible standardised Java persistence to any conceivable type of datastoreData Nucleushttp://firstname.lastname@example.com:blogger.com,1999:blog-1936942592343397099.post-55523839314021644412014-04-28T23:42:00.001-07:002014-04-28T23:46:45.557-07:00Want to assist in the development of JDO 4.0?
We have, since 2006, been reliant on the Apache JDO project to push forward the JDO standard. Politics have finally become too much for this arrangement, with Oracle involving lawyers to prohibit progress, and the Apache organisation not being as rapid as it should be at getting releases out of the door. I have now forked their API (now present in DN GitHub). The aim is for this repository to be used to develop JDO 4.0 (and maybe later dependent on people's involvement), initially to get a TypeSafe query mechanism into the JDO API (originally contributed back in April 2010!!), and bring the API up to date wrt generics in queries themselves. There are also many other features that were requested in Apache JDO JIRA over the last few years that have been simply left, but are needed.

If you, or your company uses JDO this is your chance to get involved and push it forward to cater for these required features. Do you really want to have to cast the return from your Query to be a List<candidate> etc. You could offer your real-world experience with the JDO API and what you think could be done to make it easier to use.

Should the Apache JDO project "wake up" at some point in the future we could easily enough merge our changes back into their API and discontinue this fork.
andyhttp://email@example.com:blogger.com,1999:blog-1936942592343397099.post-8562391513232952702014-04-20T08:46:00.000-07:002014-04-21T23:25:41.903-07:00Bytecode Enhancement contract in DataNucleus AccessPlatform v4.0Now in GitHub master, for DataNucleus AccessPlatform v4.0, we have changed the bytecode enhancement contract.
Since the days of JPOX we've always used the JDO bytecode enhancement contract as defined in the JDO spec. This has always been adequate to provide the necessary hooks into the object to allow for "transparent persistence". Saying that though, it does mean that anyone using DataNucleus would always have to have jdo-api.jar in their CLASSPATH, even when using JPA. This was clearly undesirable, but not a large price to pay for easy provision of JPA.

In v4.0 onwards we will enhance classes to implement org.datanucleus.enhancer.Persistable. This is very similar in terms of structure, just that methods are now prefixed "dn" instead of "jdo", and there is now a method to get the ExecutionContext that is managing the object (whereas before it was a PersistenceManager, which made very little sense for JPA usage).

Why the change?

Oracle is putting significant obstacles in the way of having further releases of the JDO standard, involving lawyers etc. Additionally, following the Apache way, the Apache JDO project has not exactly operated very efficiently in terms of getting releases out of the door. Moreover we want to remove the requirement of having to have jdo-api.jar in the CLASSPATH for JPA usage. This change will also mean that we can, in principle, improve the bytecode enhancement contract to make things more efficient or add on more information to enhance the persistence process without being restricted by what JDO has bothered to standardise.

What does this mean for a typical user?

It means very little in reality, and the majority of applications will work unchanged (apart from having to re-enhance the classes). Some minor things that will change

Wherever you use enhanced classes, you will need datanucleus-core.jar in the CLASSPATH

JPA users won't need to have jdo-api.jar in the CLASSPATH

Internally DataNucleus now uses its own builtin single-field identity classes, and if you refer to javax.jdo.identity.* classes (for JDO contexts) will auto-convert to our own class for internal use. See this package for the DN internal identity classes. There is really no need to use these JDO builtin classes directly since DataNucleus will always select the most appropriate id type when you have a single PK field.

You no longer check if a returned object is of type javax.jdo.spi.PersistenceCapable since it won't be, instead being a org.datanucleus.enhancer.Persistable.

Please register any concerns/queries in the comments section
andyhttp://firstname.lastname@example.com:blogger.com,1999:blog-1936942592343397099.post-3742007273398016512013-08-15T06:07:00.001-07:002013-12-05T06:11:08.405-08:00Configuring persistence of fields/properties using TypeConverters with JDOJPA 2.1 allows a user to specify a converter on the value of a field/property for how it is persisted in the datastore. The way we implement that in DataNucleus is to have the JPA converter as a wrapper to our own internal TypeConverter class. This means that we can make the TypeConverter mechanism available to JDO users too. Here's how it works. The first thing to do is to define a TypeConverter. As you can see, it requires implementation of just 2 methods, one for use in converting the (raw) field value into the datastore value, and one for use in converting the datastore value back into the (raw) field value. This means that the user has significant flexibility on how their values are stored, and can define their own converters and not rely on them being part of DataNucleus. If we take an example, here we want to store a field as serialised, but not standard Java serialised, instead using the Kryo library. So we define our TypeConverter as

public class KryoSerialiseStringConverter implements TypeConverter<Serializable, String>
{
 ThreadLocal kryo = new ThreadLocal();

 public Kryo getKryo()
 {
 Object value = this.kryo.get();
 if (value == null)
 {
 value = new Kryo();
 this.kryo.set(value);
 }
 return (Kryo)value;
 }

 public String toDatastoreType(Serializable memberValue)
 {
 if (memberValue == null)
 {
 return null;
 }

 Kryo kryo = getKryo();
 String str = null;
 ByteArrayOutputStream baos = new ByteArrayOutputStream();
 Output output = null;
 try
 {
 output = new Output(baos);
 kryo.writeClassAndObject(output, memberValue);
 }
 finally
 {
 output.close();
 str = new String(Base64.encode(baos.toByteArray()));
 }

 return str;
 }

 public Serializable toMemberType(String datastoreValue)
 {
 if (datastoreValue == null)
 {
 return null;
 }

 Kryo kryo = getKryo();
 byte[] bytes = Base64.decode(datastoreValue);
 Object obj = null;
 ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
 try
 {
 Input input = null;
 try
 {
 input = new Input(bais);
 obj = kryo.readClassAndObject(input);
 }
 finally
 {
 try
 {
 bais.close();
 }
 finally
 {
 if (input != null)
 {
 input.close();
 }
 }
 }
 }
 catch (Exception e)
 {
 throw new NucleusException("Error Kryo deserialising " + datastoreValue, e);
 }
 return (Serializable)obj;
 }
}
Now we need to register this converter under a name with DataNucleus runtime. Here we define a plugin.xml at the root of the plugin jar as

<plugin>
 <extension point="org.datanucleus.type_converter">
 <type-converter name="kryo-serialise" member-type="java.lang.Serializable" datastore-type="java.lang.String"
 converter-class="org.datanucleus.store.types.converters.KryoSerialiseStringConverter"/>
 </extension>
</plugin>

 So this converter is internally known as kryo-serialise. We add a MANIFEST.MF to our plugin jar as (something like)

Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: DataNucleus Converter using Kryo
Bundle-SymbolicName: org.datanucleus.store.types.converters.kryo;singleton:=true
Bundle-Version: 3.2
Bundle-Localization: plugin
Bundle-Vendor: DataNucleus
Require-Bundle: org.datanucleus
Import-Package: com.esotericsoftware.kryo;version="2.21",
 com.esotericsoftware.kryo.io;version="2.21"

just defining the dependencies of this plugin for OSGi and the plugin mechanism. Now we just need to use it on a sample class and apply the conversion to a field/property.

@PersistenceCapable
public class Sample
{
 @PrimaryKey
 long id;

 @Extension(vendorName="datanucleus", key="type-converter-name", value="kryo-serialise")
 String longString;

 ...
}
 Now whenever instances of Sample are persisted the field longString will be converted using our kryo-serialise converter. Easy! You can get the code for this example converter at DataNucleus GitHub repoandyhttp://firstname.lastname@example.com:blogger.com,1999:blog-1936942592343397099.post-5986333811798275322013-06-26T00:15:00.001-07:002013-06-27T10:56:18.004-07:00AccessPlatform 3.3 and JPA 2.1We will soon be releasing AccessPlatform 3.3. This is coming very soon after 3.2, and the reason for this is that it is simply AccessPlatform 3.2 plus full support for JPA 2.1 (i.e an upgraded datanucleus-api-jpa plugin). From that point both of these "versions" of AccessPlatform will be maintained for a period of time. So what is provided in JPA 2.1 ?

Stored Procedures
You can execute RDBMS Stored Procedures using the JPA API. This API should give you quite complete control over execution of any stored procedure, setting of IN/OUT/INOUT parameters and obtaining result set(s). To give an example

StoredProcedureQuery spq =
 em.createStoredProcedureQuery(procName);
spq.registerStoredProcedureParameter("PARAM1",
 Integer.class, ParameterMode.OUT);
boolean val = spq.execute();
Object paramVal = spq.getOutputParameterValue("PARAM1");

So here we have a stored proc that we register an Integer output parameter, and retrieve the value when we execute it. This is just one mode of operation, and you can see more usages in the documentation.

Entity Graphs
JPA has only had an equivalent of the JDO "default fetch group" since its inception. In this release it finally gets some degree of control over what fields are fetched when fetching an object from the datastore. To give an example, we have a class Person and want to pull in a field "bestFriend" under some circumstances but not by default. We define a named EntityGraph in metadata

@Entity
@NamedEntityGraph(name="includeFriend", attributeNodes= {@NamedAttributeNode("bestFriend")})
public class Person
{
 @OneToOne;
 Person bestFriend;

 ...
}
and now we want to use this entity graph when loading an object of this type. We do this as follows.
EntityGraph friendGraph = em.getEntityGraph("includeFriend");
Properties props = new Properties();
props.put("javax.persistence.loadgraph", friendGraph);
MyClass myObj = em.find(Person.class, id, props);

 So we retrieved the EntityGraph, and then used it in the find method. Equally we could have used it in a Query. You can read more about this topic in the documentation.

Schema Generation
JPA 2.1 allows generation of the schema as an up front task, or via Persistence.generateSchema(). You can specify this by making use of persistence properties, for example javax.persistence.schema-generation.database.action set to create. See the available persistence properties for details.

Foreign-Keys and Indexes
JPA 2.1 adds on the ability to specify RDBMS schema foreign-keys and indexes, for use during schema generation. By default a JPA implementation is free to generate whatever foreign-keys it decides are appropriate, but this ability allows a user to override this and control what is generated.

@OneToOne
@JoinColumn(name="BESTFRIEND_ID", foreignKey=
 @ForeignKey(name="BESTFRIEND_FK", foreignKeyDefinition=
 "FOREIGN KEY (BESTFRIEND_ID) REFERENCES PERSON (PERSON_ID) ]"))
Person bestFriend;

which will create a foreign key called "BESTFRIEND_FK" for this purposes. Similarly we can define indexes on a table of a class.

@Entity
@Table(indexes={@Index(name="FIRSTNAME_IDX", columnList="FIRST_NAME")})
public class Person
{
 @Column(name="FIRST_NAME")
 String firstName;

 ...
}
 so the firstName field is now indexed. You can read more about this topic in the documentation.

Criteria UPDATE/DELETE queries
Whilst the JPA Criteria API is overengineered and verbose it now has the ability to generate UPDATE and DELETE queries. For example

CriteriaUpdate<person> crit = qb.createCriteriaUpdate(Person.class);
Root<person> candidate = crit.from(Person.class);
candidate.alias("p");
crit.set(candidate.get(Person_.firstName), "Freddie");
Predicate teamName = qb.equal(candidate.get(Person.firstName), "Fred");
crit.where(teamName);
Query q = em.createQuery(crit);
int num = q.executeUpdate();

which will create the JPQL "UPDATE Person p SET p.firstName = 'Freddie' WHERE p.firstName = 'Fred'". You can do similar things for DELETE queries. You can read more about this topic in the documentation

Attribute Converters
By default a JPA implementation will persist a field in a datastore column of its chosen type. You can now override this to use a converter, performing the conversion from the field type to the datastore type in your converter class. The example we use in the documentation is where we have a field in our class of type URL and want to persist this as a String-type in the datastore (VARCHAR, CHAR etc).

public class URLStringConverter implements AttributeConverter<URL,String>
{
 public URL convertToEntityAttribute(String str)
 {
 if (str == null)
 {
 return null;
 }

 URL url = null;
 try
 {
 url = new java.net.URL(str.trim());
 }
 catch (MalformedURLException mue)
 {
 throw new IllegalStateException("Error converting the URL", mue);
 }
 return url;
 }

 public String convertToDatabaseColumn(URL url)
 {
 return url != null ? url.toString() : null;
 }
}

and then in our class that has a URL field we mark the field to use this converter

@Basic
@Convert(converter=URLStringConverter.class)
URL url;

 You can read more about this topic in the documentation.

JPQL FROM "ON" clauses
When joining in JPQL previously we could not add additional constraints on the join. You can now do this using the "ON" clause.

List result = em.createQuery(
 "SELECT Object(A) FROM Account A LEFT OUTER JOIN A.login L ON L.userName = 'fred'").getResultList();

JPQL "FUNCTION"
No matter what features you put in a query language some people will always want to make use of SQL functions specific to a particular datastore. With JPQL you can now do this using the FUNCTION keyword, like this

Query q = em.createQuery(
 "SELECT p FROM Person p WHERE FUNCTION('UPPER', p.firstName) = 'FRED'");

As you can see, JPA 2.1 is a minor iteration on JPA, and you can now benefit from all of these features in DataNucleus AccessPlatform 3.3

andyhttp://email@example.com:blogger.com,1999:blog-1936942592343397099.post-77831952863061847462013-04-03T00:25:00.002-07:002013-04-07T07:04:44.748-07:00What does open-source owe you?If you ever embark on a period of time writing open source (free) software you'll almost certainly come across many different attitudes in the people who choose to use this software.

There will be some people who'll actually say thanks for providing this software, that it's helped them with their project, saved them many hours of time that otherwise they would have had to spend writing something similar. The software was suitable for their project needs, so great, your effort in making the software open source has benefitted people. Feel good.

There'll be people who maybe say thanks but typically just accept the software as it is, some sort of given, and when they have a problem they take the source code and try to work out where the problem is. They may ask you questions about how the code works, or where to look to get started in resolving the problem. And some time later they may come back with a patch for their problem, so this makes the software better. Again, a good thing.

There will also be a group who take the software and use it for their projects. If a problem occurs they will report it. Their report may provide a way of reproducing the problem, or it may not. If a problem is reported with a way of being reproduced then it can be fixed when the people writing the software have some spare time to do it. Another good thing. If there is no way provided to demonstrate it then the problem report is of little use to anyone ... unless the person who has the problem is willing to get their hands dirty, get the code and fix it (with help where necessary) since only they can see it.

A final group will take the software and if a problem occurs they keep it to themselves; it's like they expect you to be aware of everything that could possibly happen with the software, the developer is a crystal ball wielder. The people who develop that software only have a certain amount of time, and they typically will use it and test it against what their own project requires. That doesn't mean that their use-cases are the same as yours. So don't expect open source developers testing to cover what you need for your project; you could contribute tests to their suite, or donate for their time to run against other datastores if this is important to you.

You may find people who ask the question "should I ditch use of your software?" when faced with a problem, something that your software doesn't cater for, or fails on. Maybe this is in some kind of "threat" sense, fix this problem or I leave? Well the answer to that is simple really. People should do what is right for their project. They've demonstrated one way or another whether they wish to contribute anything to the open source software (problem reports, testcases, patches, documentation, blogs, testimonials, donations, etc, there are many ways). If they haven't demonstrated the willingness to do anything for the project then their input won't be missed if they go off somewhere else. Do they pay the people who develop the software ? well no. Does the license of that software imply any guarantee that all problems will be fixed immediately when the toys are thrown from the pram? nope. Maybe this software is not the correct tool for their project? in which case use the correct tool for the job, and don't vent your frustration at your choices on the people who have provided something for nothing. Further to this, stick to the old addage "don't ask someone to do what you wouldn't be prepared to do yourself", didn't your mum teach you that?

People seem to have got accustomed to having an open source solution these days, and that somehow it's their "right" to have it and their "right" to have any problems found fixed. While open source (free) software gives projects a leg up in reaching their end goal more rapidly and is a great thing for software developers, open source software owes the end user nothing. Best understand this. The end user has the opportunity to do many things to contribute to that software, make it better, repay those people who put their time into developing it. The time of these people who wrote it is important to them, even if it isn't to you; at least respect that.

Some things are for sure, when you embark on writing open source software, it can be very rewarding, very beneficial if you want a way to demonstrate to potential employers of your coding skills, excellent possibilities for exploring other technologies and gaining experience, working with other people with different viewpoints, but don't go into it for the gratitude :-)

[Disclaimer : while there is such a thing as commercial open source software, providing the source code yet charging for the software, what is being discussed here is the much more common open source free software]andyhttp://email@example.com:blogger.com,1999:blog-1936942592343397099.post-39117995126133484122013-02-07T03:29:00.002-08:002013-02-09T02:11:02.022-08:00Performance - effect of various features2 years ago I made a post about performance/benchmarking, and the fact that some groups like some magic black and white "X is better than Y" (and that there is only one measure of performance so it doesn't matter what object graphs are used it will always be the same). The evidence is that they are wrong. Needless to say there will always be groups that don't share our philosophy, or don't have time to do a complete analysis (though publish their results knowing that they are incomplete and likely invalid, after all it's not their software they're maybe not presenting in a fair light). Recently we had another performance exercise. This came to the conclusion "Hibernate is better than DataNucleus, and you should really just get ObjectDB". So we're back in the territory of black and white. Yes, an OODBMS ought to be way faster than RDBMS, particular when the RDBMS has a persistence layer in front of it (and you have to pay for the OODBMS besides), but that is not the subject of this post. We'll concentrate on the former component of that conclusion.

There is nothing to add to the previous blog post in terms of correctness, we stand by all of it and nothing has been demonstrated to the contrary. This blog post simply takes the recent exercise sample and demonstrates how enabling/disabling certain features has a major impact on (DataNucleus) performance. The author of that exercise demonstrated results showing that JDO and JPA with DataNucleus were on a par in terms of performance, but below Hibernate in terms of INSERTs (anything between 1.5 and 2 times) and on a par for SELECTs (some faster, some slower but more or less the same). Since JDO and JPA are shown to be equivalent, we'll just run the exercise with JDO here, but the same is easily demonstratable using JPA (because in DataNucleus you have full control over all persistence properties and features regardless of API).

The sample data used by this case is that of 3 classes. Student has a (1-N unidirectional) List of Credit and has a (1-1 unidirectional) Thesis. We persist 100000 Students each with 1 Credit and 1 Thesis. So that's 300000 objects to be inserted, and then 100000 Students queried.

The INSERT is as follows

try
{
 pm.currentTransaction().begin();
 for (int x = 0; x < 100000; x++);
 {
 Student student = new Student();
 Thesis thesis = new Thesis();
 thesis.setComplete(true);
 student.setThesis(thesis);
 List<credit> credits = new ArrayList();
 Credit credit = new Credit();
 credits.add(credit);
 student.setCredits(credits);

 pm.makePersistent(student);
 }
 pm.currentTransaction().commit();
}
finally
{
 pm.close();
}
</credit>

and the SELECT is as follows

try
{
 Query q = pm.newQuery(
 "select from " + Student.class.getName() +
 " where thesis.complete == true && credits.size()==1");
 Collection result = (Collection) q.execute();
 ... loop through results, so we know they're loaded
}
finally
{
 pm.close();
}

So we'll run (on H2 database, on a Core i5 64-bit PC running Linux, 4Gb RAM) and vary our persistence properties to see the effect.

Original persistence properties (from original author)

optimistic=true, L2 cache=true, persistenceByReachabilityAtCommit=false, detachAllOnCommit=false, detachOnClose=false, manageRelationships=false, connectionPooling=builtin

INSERT = 120s, SELECT = 6.5s

Disabled L2 cache

Since we're persisting huge numbers of objects and it takes time to cache those, and in the original authors case Hibernate had no L2 cache enabled, lets turn the L2 cache off. So we now have

optimistic=true, L2 cache=false, persistenceByReachabilityAtCommit=false, detachAllOnCommit=false, detachOnClose=false, manageRelationships=false, connectionPooling=builtin

INSERT = 106s, SELECT = 4.0s

Why the improvement? : because objects didn't need caching, so DataNucleus didn't need to generate the cacheable form of those 300000 objects on INSERT, and 100000 objects on SELECT.

Disabled Optimistic Locking

Now instead of using optimistic locking (queue all operations until commit/flush), we allow all persists to be auto-flushed. As our exercise is bulk-insert we don't care about optimistic locking since we're creating the objects. So we now have

optimistic=false, L2 cache=false, persistenceByReachabilityAtCommit=false, detachAllOnCommit=false, detachOnClose=false, manageRelationships=false, connectionPooling=builtin

INSERT = 42s, SELECT = 4.0s

Why the improvement ? : because objects are flushed as they are encountered so we don't have to hang on to a large number of changes, so the memory impact is less. Note that we could have observed a noticeable speed up also if we had instead called "pm.flush()" in the loop after every 1000 or 10000 objects. See the performance tuning guide for that.

Use BoneCP connection-pooling

Use BoneCP instead of built-in DBCP, so we have

optimistic=false, L2 cache=false, persistenceByReachabilityAtCommit=false, detachAllOnCommit=false, detachOnClose=false, manageRelationships=false, connectionPooling=bonecp

INSERT = 42s, SELECT = 3.8s

Why the (slight) improvement ? : because BoneCP has benchmarks showing that it has less overhead than DBCP

Conclusion

As you can see, with very minimal tweaking we've reduced the INSERT time by a factor of 3, and the SELECT time by a factor of 1.7! That would equate to being noticeably faster than Hibernate in the authors original timings (for both INSERT and SELECT). Note that we already had the detach flags set to not detach anything, so they didn't need tuning (but should be included if you hadn't already looked at those in your performance tests, similarly all of the other features listed in the Performance Tuning Guide referenced above).

Does the above mean that "DataNucleus is faster than Hibernate" ? Not as such, it is in some situations and not in others. We can turn on/off many things and get different results, just as Hibernate likely can (though I'd say DataNucleus is more configurable than the majority if not all of the other persistence solutions so at least you have significant flexibility to do this with DataNucleus). In the same way we could persist other object graphs and get different results due to some parts of the persistence process being more optimised than others. One thing you can definitely say is that DataNucleus has very good performance (300000 objects persisted in 42secs on a PC, and 100000 objects queried in less than 4secs) and that performance can be significantly tuned.

The other thing that we said in the original blog post and repeat here, if you are serious about performance analysis you have to dig into the details to understand why and, as a consequence, you have an idea what to tune. You also need to assess what your application really needs to perform and what is considered acceptable performance; if you're not going to make a proper attempt at tuning a persistence solution (whether that is DataNucleus, Hibernate, or any other), best not bother at all and just use what you were going to use anyway since you don't have the time to give a fair representation (which is why we don't present any Hibernate results here, so nothing hypocritical in that).

One important thing to note is that it is extremely useful to have the ability to set many of these properties on a PersistenceManager (or EntityManager) basis (so you could have a PM just for bulk inserts and disable L2 caching, or set the transaction to not be "optimistic"). JDO 3.1 adds the ability to set persistence properties on the PersistenceManager, though DataNucleus only currently supports a minimal set there - SVN trunk now has the ability to turn off the L2 cache in a PM while have it enabled for the PMF as a whole.andyhttp://firstname.lastname@example.com:blogger.com,1999:blog-1936942592343397099.post-65426213060017155922013-01-08T08:22:00.000-08:002013-01-08T08:35:15.628-08:00Enhancing in v3.2Whilst a "final release" of version 3.2 of DataNucleus is still some way off, some important changes have been made to the enhancement process that people need to be aware of, and can benefit from.

JDO : Ability to enhance all classes as "detachable" without updating metadata
When you enhance classes for the JPA API they are all made detachable without a need to specify anything in the metadata (since JPA doesn't have a concept of not being detachable). With JDO the default is not detachable (for backwards compatibility with JDO1 which didn't have the detachment concept). In v3.2 of DataNucleus you can set the alwaysDetachable option (see the enhancer docs ) and all classes will be enhanced detachable without the need to touch the metadata; much easier than opening up every class or metadata file and adding detachable="true" !

JPA : Throwing of exceptions due to the bytecode enhancement contract
The bytecode enhancement contract requires that classes throw exceptions under some specific situations where information is either not present or not valid. These always used JDO-based exceptions before to match the JDO bytecode enhancement contract exactly. These are now changed to better suit the JPA API, and remove a need to understand JDO when using JPA.

if a non-detached field was accessed then a JDODetachedFieldAccessException was thrown; this is now changed to an (java.lang.)IllegalAccessException.

in some cases where an internal error occurred a JDOFatalInternalException would be thrown; this is now changed to an (java.lang.)IllegalStateException.

No "datanucleus-enhancer.jar", and no need of external "asm.jar"

The DataNucleus enhancer was always maintained as a separate project, but is now merged into datanucleus-core.jar and so will be available directly whenever you have DataNucleus in your CLASSPATH. Taking this further, the enhancer makes use of the excellent ASM library and in v3.2 datanucleus-core.jar includes a repackaged version of the ASM v4.1 classes internally. This means that you have one less dependency also and can do enhancement with less thinking.

PS Remember, bytecode enhancement is "evil", developers of some other persistence solution told you that back in 2003, and you should never forget it! ;-)
andyhttp://email@example.com:blogger.com,1999:blog-1936942592343397099.post-13189894775710696042012-07-29T22:44:00.002-07:002012-10-27T22:45:35.180-07:00Persistence to Neo4j graph datastoresWhilst DataNucleus JDO/JPA already supported persistence and querying of objects to/from RDBMS (all variants), ODBMS (NeoDatis), Documents (XML, Excel, ODF), Web (JSON), Document-based (MongoDB), Map-based (HBase, AppEngine, Cassandra), as well as others like LDAP and VMForce, it was clear that we didn't yet have a plugin to any of the nice new graph datastores like Neo4j. To this end, we now provide a new store plugin, supporting persistence to Neo4j.

 Usage
Just like all of the other store plugins we aim to make its usage as seamless and transparent as possible so that you, the user, has a high level of portability for your application. In simple terms you just mark your model classes with JDO or JPA metadata (annotations or XML) just as you would do for RDBMS (or any other datastore), and write your JDO or JPA persistence code in the normal way. The only difference is that the data is persisted into Neo4j transparently. I've not had time to write up a tutorial yet, but the model and persistence code would be identical to persisting to any other datastore, just that in the definition of the datastore "URL" it would be something like
datanucleus.ConnectionURL=neo4j:{my_datastore_location}

Refer to the DataNucleus docs for more details. Note that the plugin is not yet released, but is available as a nightly build for anyone wishing to give it a try

 Currently supported

Each object of a class becomes a Neo4j Node.

Supports datastore identity, application identity, and nondurable identity

Supports versioned objects

Fields of all primitive and primitive wrappers can be persisted

Fields of many other standard Java types can be persisted (Date, URL, URI, Locale, Currency, JodaTime, javax.time, plus many more)

1-1, 1-N, M-N, N-1 relation is persisted as a Neo4j Relationship (doesn't support Map fields currently)

JDOQL/JPQL queries can be performed, and the operators &&, ||, ==, !=, >, >=, <, <= are processed using Cypher, with any remaining syntax handled in-memory currently.

Support for using Neo4j-assigned "node id" for "identity" value strategy.

Checks for duplicate object identity

Embedded (and nested embedded) 1-1 fields, and querying of these fields

 Likely supported soon

Processing of more JDOQL/JPQL syntaxis in Cypher to minimise any in-memory processing

Support for backed SCO collection wrappers allowing more efficient Relationship management.

Feedback is welcome (over on the DataNucleus Forum, or below in the comments). Additionally if anyone with more experience in Neo4j who would like this plugins capabilities to be enhanced why not get involved? You contribute a few patches for example - the source code is available here, and the issue tracker is a good place to start
Enjoy! andyhttp://firstname.lastname@example.com:blogger.com,1999:blog-1936942592343397099.post-31523317482717088032012-07-11T08:36:00.004-07:002012-07-11T11:40:08.542-07:00DataNucleus AccessPlatform v3.1 coming soon ...Almost a year from the release of version 3.0 and we move close to the release of version 3.1 (due late in July 2012). So what has changed in that time ?

 Consolidation
While DataNucleus' plugin architecture is very flexible, it can lead to a large number of plugins being available. This in itself is not a bad thing but, if your application is using many features, you do have to keep track of more plugins and their versions. Version 3.1 merges the following plugins into other plugins

datanucleus-management was a plugin providing JMX capabilities to DataNucleus usage. It is now merged into datanucleus-core and is now part of a new statistics monitoring API.

datanucleus-javaxtime was a plugin providing support for the new javax.time classes that will provide a real Date/Time API for Java. This will be part of JDK 1.8 IIRC, so we have moved support for these Java types into datanucleus-core. More and more people will be using them and expecting their persistence to be seamless.

datanucleus-cache had support for an early version of the forthcoming javax.cache standardised Caching API, but the API has since changed and is reaching a level of maturity. As a result we now provide support for the latest javax.cache API in datanucleus-core, so the typical user (when javax.cache is widely implemented) will not need the datanucleus-cache plugin

datanucleus-xmltypeoracle was a plugin providing support for persisting String fields to XMLType JDBC columns for Oracle. It is now merged into the datanucleus-rdbms plugin.

As a result of these changes the typical application will only need datanucleus-core, datanucleus-api-jdo or datanucleus-api-jpa, as well as the datanucleus-{datastore} plugin of your choice in the CLASSPATH at runtime. In addition some persistence properties have more sensible defaults that will mean that more applications won't need some value setting to work optimally.

 JPA 2.1

The latest revision of the JPA spec (JSR0338) is under way, and has some new features already fleshed out. In Version 3.1 of DataNucleus we provide early access support for

Stored Procedure API : This allows users of JPA to invoke stored procedures in their RDBMS and get back output parameters and/or result sets. Obviously not applicable when using JPA with a non-RDBMS datastore.

Type Converter API : This defines a way in which a user can have a field in their Entity and wish to convert the value before it gets to the datastore (and back on retrieval). For example if you have some Java type of your own and want to persist it as a String you could define an attribute converter.

Obviously as JPA2.1 continues we will continue adding features to match their spec.

 Other New Features

Whilst some of these could be argued to deserve their own section in this blog, I list here other prominent changes in version 3.1

The REST API has had significant work, and now provides much more enhanced support for JDOQL/JPQL including order clauses etc. It also now supports use of datastore identity, bulk delete, and much more.

The enhancer will now work with JDK1.7 (and higher), using the latest version of ASM.

JTA handling with JPA is now complete

Support for nondurable identity is now provided for RDBMS, MongoDB, HBase, Excel and ODF.

You can now have any nontransactional updates persisted atomically. Previously only nontransactional persists and deletes were able to be performed atomically. This means we now have a real "auto-commit" mode of operation

The HBase plugin adds support for multitenancy, as well as obeying JDO/JPA naming strategies.

The MongoDB plugin adds support for embedded objects with inheritance, obeys JDO/JPA naming strategies, and adds support for several new query features being evaluated in the datastore.

The Excel and ODF plugins add support for JDO/JPA naming strategies.

The plugin for the Google AppEngine datastore has had a long-needed upgrade, and now works with DataNucleus v3.x. So users of that platform can get access to all of the work that has happened since 2009, finally!

 DB4O dropped!

Whilst it generally is policy to add capabilities with every release, it occasionally makes sense to remove functionality that is not considered worthwhile. Support for persisting to db4o datastores now falls under this category. As versions of db4o have been released, public APIs have changed making it hard to follow their development. Additionally Versant, the parent company of db4o, have recently released their primary object datastore with a JPA API (to add to its existing JDO API). Consequently it is felt that as Versant have done absolutely nothing to assist in the process of us providing a standards based API for their software, as they are commercial and perfectly capable of committing resource to their projects, our support is now withdrawn. It remains in DataNucleus SVN for anyone who needs such like, but no resource from this project will be directed at their (commercial) datastore.

And that's it. Maintenance of version 3.0 is now at an end (except commercial), and maintenance of version 3.1 will start once we release 3.1 as well as, at some point, the start of development for version 3.2.
andyhttp://email@example.com:blogger.com,1999:blog-1936942592343397099.post-79400093309757220002011-11-05T07:15:00.000-07:002011-11-05T10:50:45.282-07:00GAE/J and DataNucleus v3 - Part 2In the previous post we saw some initial changes to make GAE/J DataNucleus plugin work with the latest version of DataNucleus plugins. In this post we describe some further features of interest to GAE users that they weren't able to use before.

Storage Version
With v2 of the plugin it will, by default, persist using a new "storage version". In v1 of the plugin it persisted no explicit information about relations, and instead relied on doing queries for parent key to find related objects; obviously when all relations were owned then this was valid. In v2 of the plugin it persists a property in the Entity for each relation (containing the Key(s) of the related object(s)), at the owner side always. In the case of unowned relations (see below) it also will persist a property in the Entity at the non-owner side of a bidirectional relation. Obviously all existing data uses v1 of the storage version, but don't let that concern you since the plugin will check for presence of this property, and if not present then fall back to v1 behaviour to get the related objects. As entities are updated the data will be migrated to v2 storage version (a migration tool to do the job in one pass is in the works also).

Unowned Relations
By default in GAE/J all relations are owned meaning that any child objects have the parent object Key as part of their Key, and persisted as part of the same entity-group. This is obviously useful in optimising retrieval of data, but there are times when you simply want your model persisting and not have imposition of ownership. In v2 of the plugin you can have unowned relations, where each object is in its own entity-group. To define a relation like this, see the following example

@PersistenceCapable
public class A
{
 @Persistent(primaryKey="true", valueStrategy=IdGeneratorStrategy.IDENTITY)
 long id;

 @Unowned
 B b;

 ...
}

@PersistenceCapable
public class B
{
 @Persistent(primaryKey="true", valueStrategy=IdGeneratorStrategy.IDENTITY)
 long id;

 @Unowned
 @Persistent(mappedBy="b")
 A a;

 String name;

 ...
}

So when we persist an object of type A with related B it will do the following

PUT the A, generating its Key, but without property for B

PUT the B, generating its Key, and with a property referring to the key of A

PUT the A with the property referring to the key of B.

It should be noted that the @Unowned annotation is simply a shortcut for @Extension(vendorName="datanucleus", key="gae.unowned", value="true")

Be aware that if you persist unowned relations in a transaction then you will need to have multi-entity-group transactions enabled, since each object is in its own entity group.

Datastore Identity
With JDO the user has the choice of having their own primary key field (application-identity), or having the identity of the object defined for them (datastore-identity). GAE v1 only allowed application-identity. In v2 of the GAE DN plugin it also allows datastore-identity. To give an example

@PersistenceCapable
@DatastoreIdentity(strategy=IdGeneratorStrategy.IDENTITY)
public class MyClass
{
 ...
}

So with this class it will persist an Entity and its Key will use IDENTITY strategy.

Interface Fields
With v2 of the plugin you can now have fields of interface type (representing a persistable type) and persist them as you would normally do. Refer to the DataNucleus docs for how to do it (paying particular attention to the type of the field in metadata)

That's a brief summary of some of the more noteworthy improvements, and hopefully now GAE/J using JDO (or JPA) is a much more pleasant place to be, and you can refer almost directly to the DataNucleus docs for many more features now. In addition to the above changes, and in fixing various other minor bugs, the code structure has been changed quite a bit so future enhancements ought to be much more rapidly achievable
andyhttp://email@example.com:blogger.com,1999:blog-1936942592343397099.post-89908140773730234722011-08-15T08:48:00.000-07:002011-11-05T07:23:41.459-07:00GAE/J and DataNucleus v3 - Part 1
Some time ago I wrote a post about GAE/J and how it provides JDO/JPA. It had many limitations and shortcomings. Recently we have had the chance to update their DataNucleus plugin to work with version 3.0. Here are the major changes that users of that plugin will see if they build and use GAE/J DataNucleus plugin from SVN trunk.

JDOQL/JPQL : Support for methods/operators

If a user sets the query extension/hint "datanucleus.query.evaluateInMemory" then the query will be evaluated in-memory. This has an obvious drawback in terms of memory utilisation (if the number of results is large), but the big plus is that it will evaluate almost all JDOQL/JPQL syntax.

JDOQL : Support for input candidate collection

You can now specify the instances that you want to query over using query.setCandidateCollection(...). Means that you have a list of instances and can query which of them match a particular filter criteria.

Primary Key Types

Previously you could only have Long, String or Key. You can now also have long.

Plugin package naming

Now uses com.google.appengine.datanucleus as its package root, hence not using the DataNucleus-owned domain.

JDOQL/JPQL setResultClass
This is now supported for the standard types of result classes, so you no longer need to convert manually the result into your required type.

Value Generation
GAE/J users can now make use of other DataNucleus value generators, such as "uuid", "uuid-hex"

PersistenceManagerFactory
The PersistenceManagerFactory used is now the standard DataNucleus PMF, not any custom GAE variant. To be specific the PersistenceManagerFactoryClass is now org.datanucleus.api.jdo.JDOPersistenceManagerFactory. If you want to have a singleton PMF, simply set the persistence property datanucleus.singletonPMFForName to true. This will then return any existing PMF if present for the requested persistence-unit, or create it if not present.

EntityManagerFactory
The EntityManagerFactory used is now the standard DataNucleus EMF, not any custom GAE variant. To be specific the PersistenceProvider is org.datanucleus.api.jpa.PersistenceProviderImpl. If you want to have a singleton EMF, simply set the persistence property datanucleus.singletonEMFForName to true. This will then return any existing EMF if present for the requested persistence-unit, or create it if not present.

JPA2
By using DataNucleus v3 you now have available all of the changes made in JPA2, so things like Criteria queries, metamodel, etc.

JDO3
By using DataNucleus v3 you now have available all of the changes made in JDO3.0/JDO3.1. This means query timeouts, metadata API, enhancer API, as well as the DataNucleus proposal for Typesafe JDO queries.

Level2 Caching
Level2 Caching is enabled by default, using an internal map-based cache. You can improve this further by setting the persistence property datanucleus.cache.level2.type to "javax.cache" and include datanucleus-cache.jar in your CLASSPATH. This will then cache using GAE Memcache

Non-transactional Persistence
DataNucleus non-transactional behaviour is different now, with any call to pm.makePersistent, pm.deletePersistent, em.persist, em.merge, em.remove being atomic, sent to the datastore immediately. Any updates to fields via setters are still queued.

JPA RetainValues
JPA usage, by default, has datanucleus.RetainValues set to true now. This means that when you commit a transaction the object will retain the values of its fields (previously it migrated to hollow state).

Persistence of other java types
In GAE/J v1 you can only persist fields of the following types : primitive, primitive wrapper, String, Date, Enum, BigDecimal, some com.google.appengine types, as well as Collection types. With v2 you can now persist fields of types Currency, Locale, Timezone, BigInteger, Color, Point, StringBuffer, Jodatime, javax.time, and many more.

Be aware though ... there is more to come
andyhttp://email@example.com:blogger.com,1999:blog-1936942592343397099.post-55034333145386601692011-03-03T04:17:00.000-08:002013-01-28T08:10:55.787-08:00Performance, benchmarkingEvery so often some individual or group decides they're going to "invent" a new benchmark. They ignore all that have been done before (like PolePos) maybe due to NotInventedHereSyndrome, or maybe due to "oh someone who helped write that was related to some datastore therefore it must be dodgy". While it is highly likely that existing benchmarks don't include their particular specific case, they never make any reference to what was there before that people are familiar with. They make minimal efforts at configuring other persistence solutions other than what is their favourite or what they have experience with, and then publish their results with a flourish to websites.

Such benchmarks have been known to draw conclusions that you have a standard performance difference across all types of tests. Anyone who has ever looked at the persistence process would know that this conclusion is flawed since good persistence solutions all provide particular features, and by turning on these features you get beneficial things at the expense of some performance, and so certain operations will be better with one tool than with another. Hibernate has some good features, DataNucleus has some good features. If you enable particular features you get poorer performance in other areas.

The other aspect of their conclusion is that they simply want a headline grabber black and white this is better than that. They seemingly aren't interested in thinking about the different methods employed by the software under test and the particular options. If I was exploring the topic of performance (and its an interesting topic, that can be useful in influencing priorities) I'd want to think about what I was asking the software to do and, bearing in mind that these benchmarks use open source software, and I'd have information available as to how a particular implementation attempts to do something. This could then be termed constructive benchmarking.

Recently we had one which took a flat class, no inheritance or relationships, and persisted it, ... many times ... and then called itself a "Detailed Comparison". While this may be an operation that an application may need to do, the response time for the persist of an object of this form would not differ by any perceptible amount to the end user of that software. If a user is wanting to persist a large number of objects (like in bulk loading), then this would typically be performed in a background task anyway. DataNucleus has never been optimised for such a task (it would logically be better at complex object graphs due to the process it uses for detection of changes), but even with it, is actually configurable to give pretty good performance. As above, it's good at some things and less good at others. Anyone claiming to make a "detailed comparison" would have bothered to try it on a range of cases, to look at the full capabilities, to look at outstanding issues etc.

Here, to give you something to talk about. I just ran a flat class

@Entity
public class A
{
 @Id
 Long id;

 String name;

 public A(long id, String name)
 {
 this.id = id;
 this.name = name;
 }
}
I ran it through the following persistence code

EntityManager em = emf.createEntityManager();
EntityTransaction txn = em.getTransaction();
txn.begin();
for (int i=0;i<10000;i++)
{
 A a = new A(i, "First");
 em.persist(a);
}
txn.commit();

I used the config of the aforementioned (Hibernate) users for running Hibernate, and tuned DataNucleus myself. And the runtimes ? Hibernate (3.6.1.Final) took 8164ms, and DataNucleus (SVN trunk on 3/Mar/2011) took 7227ms (using HSQLDB 184.108.40.206 embedded). So on that case DataNucleus was faster. Is this significant ? Well no, since as I already explained it's a particular case, but demonstrates the principle clearly enough. We can all turn on/off particular options and get some results. Besides which persisting 10000 objects in just 7-8 seconds (on this PC!) is pretty impressive in anyone's book and never a bottleneck in a normal application.

If you are performing something of the nature of a flat object bulk persist you would not use transactions to avoid the overhead, and you would turn off various features that are not of interest - managed relationships, persistence-by-reachability-at-commit, L2 caching even. Then if using an identity generator you would allocate large blocks of values with your metadata. That said, if you are so serious about performance and persisting flat objects to RDBMS then anyone sensible would use either JDBC direct, or a JDBC wrapper like MyBatis or SpringJDBC. This is "right tool for the job". Since the benchmark was provided by a group of Hibernate users (they provide various Hibernate tutorials and nothing for any other persistence implementation) we only have to assume that this must be what they think is the "best tool" - if so why not reproduce your benchmark with well-written JDBC and let us know what you find. I even posed this question about applicability of a flat class on their blog and my comment was deleted. Whether it was deleted by them, or by the blogger system I've no idea, but it was there for an hour and at that point was deleted. Our blog has never lost comments, and it's on the same host system.

One of the (undeleted) comments on their benchmark was from the author of an ODBMS who seems to like to decide publically what I ought to spend my spare time on. Is he a commercial client ? no. Is his software open source ? no. Or free ? not if I want to use it on anything serious. The response to him is simple : let me decide what I spend my time on, and you concentrate on your own software; I don't tell you what to do. It's an ancient custom called "respect" - sadly symptomatic of the attitudes in the IT profession.

A benchmark to use as the basis for choosing what software you use in your own application needs to cover the different persistence operations that you will perform. If you have a web application that creates a few objects, deletes a few, updates a few, etc continually, then the likelihood is that the performance will not impact on you or your end users to one iota. What will impact on you is whether the persistence solution allows you to do what you want to do or whether it has a large number of unfixed bugs that force you to continually implement workarounds or compromises on your design. Why not have a look through the issue tracker of the software and see what types of problems people are having, and how long the issues have existed?

Edit : to give an example of another benchmark for JPA providers, here is one that was presented in 2012 comparing the 4 most well-known JPA providers on some more complicated models. DataNucleus comes out very well. Note that in this example the author actually bothered to investigate what was happening under the covers.

DataNucleus 3.0 development is initially focussing on architecture, since we believe in getting the architecture right first to take the software to the level we think it needs to be. This means that in early milestones (the benchmark referenced above decided to use 3.0M1) we spend time on refactoring etc, and not on performance. This doesn't mean that performance isn't important, just that we feel our users want to be able to perform their tasks first and foremost and then speed things up later. This is the same methodology employed by PostgreSQL to much success, who for years had to listen to "MySQL is faster" comments. Even with that general philosophy anyone using current SVN code would already see some very noticeable speed up in non-transactional persistence performance, and anyone using the MongoDB plugin would also see much more optimised insert performance. These benefits are due to extending the architecture to do some things that we've wanted to do for some time but didn't since we wanted to maintain backwards compatibility and due to resourcing.

Next time you look at some "performance benchmark" we suggest that you bear this in mind. We won't be spending our time analysing their results, or responding to their claims. This is because we'd rather spend it developing this software, rather than "mine is better than yours" negative mentality discussions.andyhttp://firstname.lastname@example.com:blogger.com,1999:blog-1936942592343397099.post-30554882876997468952011-02-13T02:19:00.000-08:002011-02-16T10:35:15.152-08:00DataNucleus v3 and MongoDBThere has obviously been a recent shift to look at highly scalable datastores for use in "the cloud". Google wrote a plugin for their own BigTable datastore back in 2009, providing access to some of the features of JDO/JPA. Unfortunately they didn't have the intention of providing a full and fair reflection of those persistence specifications, and so reaction to it was mixed. Some people attempted to argue that APIs like JDO did not match these types of datastores (I see nothing in the API or query language of JDO that led to this conclusion, but anyway) and that using standard APIs on them was inappropriate; they were asked to provide concrete examples of features of these datastores could not be handled adequately by JDO but unfortunately didn't come up with anything.

With DataNucleus v3 we have the opportunity to spend some time on providing good support for these types of datastores, adding support for missing features. A previous blog post documented efforts to upgrade the support for HBase. In this blog post we describe the features in the new plugin for the MongoDB document-based "NoSQL" store.

Features that this plugin currently supports include

Support for single MongoDB instances, and for MongoDB replica sets

Support for application identity (defined by the user), and datastore identity (surrogate field in the JSON document)

Basic persistence/update/delete of objects

Support for persistence of (unembedded) Collections/Maps/arrays by way of storing the identity of any related object(s) in the field.

Persistence of related objects (1-1/N-1) as "flat" embedded, where all fields of the related object are fields of the owner JSON document. This also supports nested related objects (unlimited depth).

Persistence of related objects (1-1/N-1) as nested embedded, where the related object is stored as a nested JSON document within the owner JSON document. This also supports nested related objects (unlimited depth).

Persistence of related collections/arrays (1-N) as nested embedded, where the related objects are stored as an array of nested JSON documents. This also supports nested relations.

Persistence of related maps (1-N) as nested embedded, where the related map is a nested array of map entries with fields "key","value". Supports nested relations.

Persistence of fields as serialised.

Polymorphic queries. If the query requests a class or subclasses then that is what is returned. This implies the execution of any query against the MongoDB collection for each of the possible candidate classes.

Access to the native MongoDB "DB" object via the standard JDO datastore connection accessor

Support for persistence of object version, stored in a separate field in the JSON document and support for optimistic version checking

Support for "identity" value generation using the MongoDB "_id" field value. The only restriction on this is that a field/property using "identity" value generation has to be of type String

Support for "increment" value generation (numeric fields).

Support for SchemaTool creation/deletion of schemas. This supports the document collection for the classes, as well as any indices required (including unique).

JDOQL/JPQL querying, including support for fetch groups, so you can restrict how much data is returned by the query.

Basic JDOQL/JPQL filter clauses (comparison operations) are evaluated in the datastore where possible.

Support for running queries on MongoDB slave instances.

Support for persistence of discriminators, so a user can store multiple classes into the same MongoDB collection, and we use the discriminator to determine the object type being returned.

As you can see, we already provide a very good JDO (and JPA) capability for MongoDB, and the feature list is shown as a matrix here.

Input to this plugin is obviously desired, particularly from people with more intimate knowledge of MongoDB. Source code can be found at SourceForge, and issues are tracked via the NUCMONGODB project in JIRA.andyhttp://email@example.com:blogger.com,1999:blog-1936942592343397099.post-6681089051277975232011-01-24T06:57:00.001-08:002011-01-29T06:01:33.826-08:00JPA : TCK request and JPA2.1It is now almost exactly a year since we submitted a request for access to the JPA2 TCK (JSR0317). We provided everything requested of us by Oracle, and we still haven't received the JPA2 TCK. Just to say Happy Anniversary Oracle on this evidently controversial request. The only possible reasons that I can think of for lack of provision of this are either incompetence, or deliberate prevention of access. Needless to say, we no longer wish to have access to this TCK.

On a related note I was asked whether I'd be participating in the JPA2.1 (JSR0338) expert group. I did think about it, no really! :-) But then I remembered that I spent many hours sending emails chasing Oracle employees to fulfil the request for the JPA2.0 TCK all to no avail. We also have to remember that JPA was born out of politics nothing more. Consequently the answer had to be a NO.

I have every willingness to participate in truly open standards on Java persistence, based on technical reasons. Any such participation would be with open mailing-lists/forums, and an open testing capability so that anyone can validate the compliance of any implementation. JPA, and the JCP process it operates under, is not that; JPA standardises things like "Criteria" that is widely recognised as inelegant (many call it ugly actually) and hard to use. A standard should take something that has become accepted as a good way of doing something (such as QueryDSL, for example) and convert it into something coherent with a clean API (like what we did with JDO Typesafe); it should not invent something and impose it on people. The JPA philosophy is simply wrong in this respect. I wish them good luck with JPA2.1, and DataNucleus will, at some point (typically as soon as the spec is made public), implement what they come up with, but I will not be party to its "design".andyhttp://email@example.com:blogger.com,1999:blog-1936942592343397099.post-43454949539205588382011-01-13T10:40:00.000-08:002011-01-20T08:17:02.631-08:00DataNucleus v3 and HBaseDataNucleus AccessPlatform v3 provides an opportunity to bring some of our other datastore plugins closer to the standard of the more mature long-supported datastores (e.g RDBMS). In the case of HBase, the plugin provided with v2.0 offered basic persistence and querying, at best. In v3 this is already much improved.

You can now run SchemaTool against HBase. This operates in either "create" or "delete" modes, and allows you to manage the schema required by your persistable classes.

When a relationship was persisted with v2.0 it simply serialised the related object. This broke JDO/JPA cascade rules. In v3 the column for the relation in the owner stores the identity (or identities when persisting multi-value relations). This also provides correct cascading of persist and update.

When a String field was persisted in v2.0 it was Java serialised, meaning that it was not readable with something like "hbase shell". In v3.0 String/char fields are persisted as the bytes of the field value, hence readable.

With v2.0 we provided value generation using "uuid", "uuid-hex", etc simple generators, but not accessible by default. In v3.0 the default (JDO "native", JPA "auto") is "uuid-hex", and we also provide an "increment" generator (contrib from Peter Rainer).

In v2.0 we only supported "application identity". In v3.0 we also support "datastore identity" (surrogate identity column).

In v2.0 we didn't support storing a version against the object. In v3.0 we do allow this.

You can now embed persistable fields into the table of the owning object, and also nested embedded fields.

Now supports HBase 0.90

Obviously there is still much more that can be done - see the datastore features comparison table. One that will give more performance is to handle more of any query filter in the datastore, rather than just processing all in-memory. Contributions for this and other things are very welcome. But then we aren't even at 3.0 Milestone 1 yet ...andyhttp://email@example.com:blogger.com,1999:blog-1936942592343397099.post-88793233184373119292010-11-08T23:24:00.000-08:002011-03-05T00:35:50.624-08:00JDO Typesafe .vs. JPA CriteriaJPA2 introduced its "Criteria" queries, providing an API for typesafe query generation without the need to hardcode field names etc in queries; it built on the approach of Hibernate Criteria. DataNucleus includes a proposal for JDO "Typesafe" queries. It takes a slightly different approach aiming at usability and elegance. In this blog post we compare some queries using the two APIs.

In these examples we have two classes, Inventory and Product, where Inventory has a set of products.

Select of persistable objects with simple filter

JPQL single-string would be

SELECT p FROM Product p WHERE p.name = 'MP3 Extra'

whilst JDOQL single-string would be

SELECT FROM Product WHERE this.name == 'MP3 Extra'

JPA Criteria would be

CriteriaQuery<Product> criteria = builder.createQuery(Product.class);
Root<Product> productRoot = criteria.from(Product.class);
criteria.select(productRoot);
criteria.where(builder.equal(productRoot.get(Product_.name), "MP3 Extra"));
List<Product> products = em.createQuery(criteria).getResultList();

JDO Typesafe is

TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
List<Product> results = tq.filter(cand.name.eq("MP3 Extra")).executeList();

Select of result of attributes of persistable objects

JPQL single-string would be

SELECT p.value, p.manufacturer FROM Product p WHERE p.name = 'MP3 Extra'

JDOQL single-string would be

SELECT this.value, this.manufacturer FROM Product WHERE this.name == 'MP3 Extra'

JPA Criteria would be

CriteriaQuery criteria = builder.createQuery();
Root<Product> productRoot = criteria.from(Product.class);
criteria.multiselect(productRoot.get(Product_.value),
productRoot.get(Product_.manufacturer);
criteria.where(builder.equal(productRoot.get(Product_.name), "MP3 Extra"));
List<Object[]> results = em.createQuery(criteria).getResultList();

JDO Typesafe is

TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
List<Object[]> results =
tq.filter(cand.name.eq("MP3 Extra"))
.executeResultList(cand.value, cand.manufacturer);

Select of aggregate of attribute of persistable objects

JPQL single-string would be

SELECT MAX(p.value) FROM Product p WHERE p.name = "MP3 Extra"

JDOQL single-string would be

SELECT MAX(this.value) FROM Product WHERE this.name == "MP3 Extra"

JPA Criteria would be

CriteriaQuery<Integer> criteria = builder.createQuery(Integer.class);
Root<Product> productRoot = criteria.from(Product.class);
criteria.select(builder.max(productRoot.get(Product_.value)));
criteria.where(builder.equal(productRoot.get(Product_.name), "MP3 Extra"));
Object result = em.createQuery(criteria).getSingleResult();

JDO Typesafe is

TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
Integer result =
tq.filter(cand.name.eq("MP3 Extra")).executeResultUnique(Integer.class, cand.value.max());

Select of persistable objects with simple filter and parameter

JPQL single-string would be

SELECT p FROM Product p WHERE p.name = :param

whilst JDOQL single-string would be

SELECT FROM Product WHERE this.name == :param

JPA Criteria would be

CriteriaQuery<Product> criteria = builder.createQuery(Product.class);
Root<Product> productRoot = criteria.from(Product.class);
criteria.select(productRoot);
ParameterExpression<String> valueParam = builder.parameter(String.class);
criteria.where(builder.equal(productRoot.get(Product_.name), valueParam));
TypedQuery<Product> query = em.createQuery(criteria);
query.setParameter(valueParam, "MP3 Extra");
List<Product> products = query.getResultList();

JDO Typesafe is

TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
List<Product> results =
tq.filter(cand.name.eq(tq.stringParameter("prefix")))
.setParameter("prefix", "MP3 Extra").executeList();

Select of persistable objects with joined filter condition

JPQL single-string would be

SELECT i FROM Inventory i JOIN i.products p WHERE (p.name = 'MP3 Extra')

JDOQL single-string would be

SELECT FROM Inventory WHERE this.products.contains(var) && var.name == "MP3 Extra"

JPA Criteria would be

CriteriaQuery<Inventory> criteria = builder.createQuery(Inventory.class);
Root<Inventory> invRoot = criteria.from(Inventory.class);
criteria.select(invRoot);
Join<Inventory, Product> productJoin = invRoot.join(Inventory_.products);
criteria.where(builder.equal(productJoin.get(Product_.name), "MP3 Extra"));
List<Inventory> inventories = em.createQuery(criteria).getResultList();

JDO Typesafe is

TypesafeQuery<Inventory> tq = pm.newTypesafeQuery(Inventory.class);
QProduct var = QProduct.variable("var");
QInventory cand = QInventory.candidate();
List<Inventory> results =
tq.filter(cand.products.contains(var).and(var.name.eq("MP3 Extra"))).executeList();

Select of persistable objects with subquery filter

JPQL single-string would be

SELECT p FROM Product p WHERE p.value < (SELECT AVG(q.value) FROM Product q)
JDOQL single-string would be

SELECT FROM Product WHERE this.value < (SELECT AVG(q.value) FROM Product q)
JPA Criteria would be

CriteriaQuery<Product> criteria = builder.createQuery(Product.class);
Root<product> productRoot = criteria.from(Product.class);
criteria.select(productRoot);
Subquery<Double> sub = criteria.subquery(Double.class);
Root<Product> subRoot = sub.from(Product.class);
criteria.where(builder.lt(productRoot.get(Product_.value),
sub.select(builder.avg(subRoot.get(Product_.value)))));
List<Product> products = em.createQuery(criteria).getResultList();</product>
JDO Typesafe is

TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
TypesafeSubquery<Product> tqsub = tq.subquery(Product.class, "q");
QProduct candsub = QProduct.candidate("q");
List<Product> results =
tq.filter(cand.value.lt(tqsub.selectUnique(candsub.value.avg()))).executeList();

As you have seen, for all typesafe queries JDO Typesafe is more elegant and shorter. It requires no "builder" too. Access of a field of a class is one particular example, typing "cand.value" instead of "productRoot.get(Product_.value)"andyhttp://email@example.com:blogger.com,1999:blog-1936942592343397099.post-83900930737490434332010-11-02T03:07:00.000-07:002010-11-09T01:31:27.487-08:00JDO Typesafe Queries : Part 3 - ExamplesIn two previous blog posts, we introduced the idea of having a typesafe refactor friendly API for queries for JDO, and then described typical expression types necessary to achieve that. In this blog post we take that a step further, particularly now that DataNucleus SVN has a proof of concept query API to fulfil this requirement.

In these examples we have a class Product. Consequently we have a "query" class autogenerated - QProduct. Here is the QProduct class

public class QProduct
 extends PersistableExpressionImpl<Product>
 implements PersistableExpression<Product>
{
 public static QProduct candidate() {...} // candidate "this"
 public static QProduct candidate(String name) {...}
 public static QProduct parameter(String name) {...}
 public static QProduct variable(String name) {...}

 public NumericExpression<Long> id;
 public StringExpression name;
 public NumericExpression<Double> value;

 ...
}

So, as you can see we have access to the persistable fields/properties of Product by way of public fields in its query class. Now on to some examples

Filter only
Here we select all Products that have a value less than 40.0

TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
tq.filter(cand.value.lt(40.0));
List<Product> results = tq.executeList();

Filter + Order
Here we select all Products that have a value less than 40.0 and ordering by their name

TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
tq.filter(cand.value.lt(40.0)).orderBy(cand.name.asc());
List<Product> results = tq.executeList();

Filter + Order + Result
Here we select the product name and product value for all Products that have a value less than 40.0 , ordering by their name

TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
tq.filter(cand.value.lt(40.0)).orderBy(cand.name.asc());
List<Object[]> results = tq.executeResultList(true, cand.name, cand.value);

Filter with Methods
Here we select all Products that have a value less than 40.0 and name starting with "Wal", and ordering by their name

TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
tq.filter(cand.value.lt(40.0).and(cand.name.startsWith("Wal")));
List<Product> results = tq.executeList();

Filter, using Parameters
Here we select all Products that have a value less than 40.0, using a parameter

TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate()
tq.filter(cand.value.lt(tq.doubleParameter("param1"))).setParameter("param1", 40.0);
List<Product> results = tq.executeList();

Order + Range
Here we select Products, ordering by the name and restrict to the first two

TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
tq.orderBy(cand.name.asc()).range(0,2);
List<Product> results = tq.executeList();

Filter with Variable
Here we select Inventory (which has a collection of Products) that contain a Product with a particular name

TypesafeQuery<Inventory> tq = pm.newTypesafeQuery(Inventory.class);
QInventory cand = QInventory.candidate();
QProduct var = QProduct.variable("var");
tq.filter(cand.products.contains(var).and(var.name.startsWith("Wal")));
List<Inventory> results = tq.executeList();

Filter with Subquery
Here we select Products with a value less than the average value of any Product

TypesafeQuery<Product> tq = pm.newTypesafeQuery(Product.class);
QProduct cand = QProduct.candidate();
TypesafeSubquery tqsub = tq.subquery(Product.class, "p");
QProduct candsub = QProduct.candidate("p");
tq.filter(cand.value.lt(tqsub.select(candsub.value.avg())));
List<Inventory> results = tq.executeList();

As you can see from the above queries we have no hard-coded class or field names, providing better refactorability. We can obviously make use of all JDOQL supported methods in the above queries, and define parameters and variables just like in the current JDOQL API.

It should be noted that this JDOQL facility is much more elegant and requires less code than the equivalent queries using JPA "Criteria".andyhttp://email@example.com:blogger.com,1999:blog-1936942592343397099.post-50980892025319650742010-07-14T11:05:00.000-07:002010-07-15T03:21:48.299-07:00Writing data to Excel from Java - the simple wayOn the web you can find some blogs about how to use Apache POI to create an Excel spreadsheet. They typically contain reams of code just to add a simple table and some rows of data. It's always amazing the amount of suffering people put themselves through to perform simple tasks. An example is this blog (there are others) which demonstrates some typical calls you will make using Apache POI to create a simple spreadsheet. Whilst that may represent what Apache POI requires, a user choosing to follow this way is inflicting unnecessary suffering on themselves, really. Read on ...

Here's how we would achieve the same thing as the referenced blog using JDO and DataNucleus. Firstly lets create a class that equates to a row of data in the table.

@PersistenceCapable(table="SampleDataSheet1")
public class SampleData
{
 @PrimaryKey @Column(name="Employer Name")
 String employerName;
 @Column(name="Designation")
 String designation;
 @Column(name="Country")
 String country;

 public SampleData(String e, String d, String c)
 {
 this.employerName = e;
 this.designation = d;
 this.country = c;
 }
}

Now we define our datastore persistence properties (datanucleus.properties)

javax.jdo.PersistenceManagerFactoryClass=
 org.datanucleus.jdo.JDOPersistenceManagerFactory
javax.jdo.option.ConnectionURL=excel:file:test.xls

Finally let's do some persistence

PersistenceManagerFactory pmf =
 JDOHelper.getPersistenceManagerFactory("datanucleus.properties");
PersistenceManager pm = pmf.getPersistenceManager();
Transaction tx = pm.currentTransaction();
tx.begin();
SampleData ds =
 new SampleData("Intelligent User", "Software Engineer", "Bolivia");
pm.makePersistent(ds);
tx.commit();

and this generates the same output that was shown in the blog, just that the code is now readable, shorter, and uses standardised APIs, never mind the fact that you could conceivably just persist the same data to RDBMS, XML, ODF, ODBMS, Cassandra and many more datastores with a simple change of that datastore URL. Additionally it allows you to focus on objects, which is likely why you chose Java in the first place. Why put yourself through more pain ?

[Needless to say, DataNucleus would also allow you to do the same using JPA]andyhttp://firstname.lastname@example.com:blogger.com,1999:blog-1936942592343397099.post-10202066048874693852010-07-13T11:51:00.001-07:002010-07-13T11:51:32.471-07:00TJDO has left the buildingAs anyone who has been using DataNucleus (and JPOX before it) for some time will know, it all started out (in 2003) as TJDO. This project was forked to provide all of the missing functionality that JDO requires and users of that technology demand (starting with application identity, and all common ORM relationship patterns). Over the last 7 years components have been rewritten one-by-one

Table handling code was upgraded to cater for JDO2 inheritance strategies

MetaData was rewritten to match the JDO2 definition (replacing TJDO's JDO1-style handling)

Transaction handling was rewritten to cater for XA transactions

Mapping code was rewritten to split into java type handling and datastore handling

DataNucleus SVN trunk (v2.2) now removes the last significant component of TJDO ... its JDOQL implementation for RDBMS (known in v2.1 as "Legacy"); it is now totally replaced by the generic-query-expression-tree approach that allows many many additional JDOQL queries to execute. There still remain some TJDO vestiges but nothing of any significance. R.I.Pandyhttp://email@example.com:blogger.com,1999:blog-1936942592343397099.post-57166303601516395492010-07-07T10:43:00.000-07:002010-07-12T02:08:50.959-07:00JDO Typesafe Queries : Part 2 - ExpressionsContinuing on from the previous post, in order to be able to express components of a query in a Java style we need to represent all fields/properties/parameters/variables as expressions. The type of the field/property/parameter/variable determines its expression type. Let's take an example

public class Person
{
 String firstName;
 String lastName;
 int age;
 Person bestFriend;
}

We need to represent these field types with expressions, so let's start with

StringExpression - for Strings

NumericExpression - for int, long, short, double, float, Integer, Long, Short, Double, Float, BigDecimal, BigInteger etc

BooleanExpression - for boolean, Boolean

ByteExpression - for byte, Byte

CharacterExpression - for char, Character

DateExpression - for Date-based types

ObjectExpression - for other Object-based types where we have no specific handling

Ok, so this is all well and good and we can express firstName as a StringExpression, and similarly age is a NumericExpression. So for that we can do

QPerson person = QPerson.person;
Query<Person> q = pm.newTypesafeQuery(person);
Person bob =
 q.filter(person.firstName.eq("Bob")).executeUnique();

To represent a persistable field (i.e a 1-1 relation, bestFriend in the above example) we have another expression type PersistableExpression.

"Q" classes

Above you see use of metamodel query classes. We refer to them as "Q" classes currently, but the naming is arbitrary for now. So what does a "Q" class look like ?

public class QPerson implements PersistableExpression<Person>
{
 public static final QPerson person = new QPerson("person");

 public final QPerson bestFriend;
 public final NumericExpression age;
 public final StringExpression name;

 ... (implementation of other PersistableExpression methods).
}

So in simple terms, we have a public field for each of the normal fields in Person, but that are of XXXExpression types. So when a user accessed QPerson.person they get the candidate for use in a query. Then they can do "person.name" and this is a StringExpression. The NumericExpressionImpl/StringExpressionImpl are the implementations for the particular provider of this typesafe query (e.g DataNucleus). You also notice above that the bestFriend field is also a QPerson, and hence a PersistableExpression, so we can chain field access as "person.bestFriend.firstName"

Methods of field types

Obviously in JDOQL (and Java), we allow some method calls. This is represented here by adding the supported methods to StringExpression, NumericExpression etc. For example StringExpression has a method toUpperCase() to match what Java allows, so we can upgrade the query example to be

QPerson person = QPerson.person;
Query<Person> q = pm.newTypesafeQuery(person);
Person bob =
 q.filter(person.firstName.toUpperCase().eq("BOB")).executeUnique();

There are still some more challenging areas of the API to work out, but the above is just to give a further taster and provoke comment.

Please refer to the current javadocs here (Work-in-progress).andyhttp://email@example.com:blogger.com,1999:blog-1936942592343397099.post-5082258995931888542010-06-22T07:29:00.000-07:002010-06-22T09:08:18.390-07:00JDO Typesafe Refactorable QueriesOne of the primary drawbacks with current JDO queries is that they are string-based, so you are hardcoding field names etc. When you refactor a (model) class, the field names are left unchanged, so the query no longer works. Some efforts have been made to simplify this process by providing a form of type-safe query (called "criteria" queries by some, but the name is misleading since all queries define criteria) - see JPA2 Criteria queries, and also QueryDSL. Of the current offerings the QueryDSL fluent API approach is considered by many to be the most elegant.

JDO needs such a type-safe refactorable query API. However it is important that this does not go down the track followed by JPA Criteria queries (IMHO) since it leads to excessive code just to generate a query, hence the fluent approach make more sense. What we are proposing would be something similar to the following (taken from QueryDSL, but adapted to use more JDOQL-like syntax)

QCustomer customer = QCustomer.customer;
Query<Customer> q = pm.newTypesafeQuery(customer);
Customer bob = q.filter(customer.firstName.eq("Bob"))
 .unique(true).execute();

where the QCustomer class is generated automatically (during compilation via an annotation processor, or optionally via a separate process when not using JDO annotations).

The idea is to provide support for all JDOQL syntax via this API, yet provide type-safe handling and refactorability, introducing this in the DataNucleus AccessPlatform 2.2 timeframe, and finally to standardise something similar into JDO 3.1. Your ideas/comments are welcome.andyhttp://email@example.com:blogger.com,1999:blog-1936942592343397099.post-66585932653446787972010-06-03T05:36:00.000-07:002010-06-18T03:28:24.406-07:00Persisting multisets (bags) with DataNucleusA multiset is a set where you can have a particular object multiple times in the set (a normal java.util Set will remove dups). This is also known by some as a "bag". Google's Collections library has such a container class (Multiset), and fields of this type are now persistable using DataNucleus.

Put the datanucleus-googlecollections plugin in the CLASSPATH

Develop your model using the Multiset java type.

Make sure that your Multiset field uses a join-table when persisting to RDBMS, and that the join-table doesn't have a primary-key defined

So if we have

@PersistenceCapable
public class Sample
{
 @Join(extensions=
 @Extension(vendorName="datanucleus",
 key="primary-key", value="false"))
 Multiset<Element> elements = null;
 ...
}

this will create an RDBMS join table as

CREATE TABLE SAMPLE_ELEMENTS
(
 ID_OID BIGINT NOT NULL,
 ID_EID BIGINT NOT NULL
);

i.e no primary key, and so we can have duplicates of the same element in the Multiset.
Thereafter the persistence just performs as normal (for JDO or JPA).andyhttp://email@example.com:blogger.com,1999:blog-1936942592343397099.post-42381945680910724772010-04-16T08:16:00.001-07:002010-06-22T11:27:33.668-07:00JPA, and the role of complianceFor some time we've been trying to get our hands on the JPA2 TCK (JSR317). This is a suite of tests that is supposed to define if your JPA implementation is "compliant". JPA (1, 2) are developed in private (secret) by the JCP. You can try to get on their "Expert Group" but since JPA1 did not allow in "just anybody" (and people were hand-picked in those days), there seems little point in participating in that.

To actually get your hands on the "TCK" for any JSR you need to sign some form and send it to Sun/Oracle by snail mail. Since I sent one of these forms in (for the JPA1 TCK) back in 2007 then it really ought not to be necessary one would think. Well that's what we were advised by SUN/Oracle also (after a wait of a month and half). So here we are more than 2 months on and still no TCK. Not even a sniff of one.

Why is this TCK so secret ?
This seems a reasonable question. What is so sensitive in it that it shouldn't be open to everyone ? After all, with an open TCK, anybody would be able to actually test the implementation of their choice and verify if it is indeed compliant. They would also be able to view the testcases and contribute new ones that were missed in the test suite to cover other functionality. Obviously not everyone would want to download it and check people's claims, but then some would and it would hold people's claims up for review, unlike now. Really, why do I have to sign an NDA just to see it ? and why can't I publish the results in full on our website ?

Why does it take so long ?
Now granted SUN is merging into Oracle, but really 2+ months doesn't really seem too efficient. Why is the process manual, snail-mail driven ? After all, this is something electronic at the end of the day, so you would have expected that if people really have to apply for something then there would be a page on which to request it, and it would forward the infos to some team who approve or otherwise. Is this something typical of US organisations in general, or just Oracle?

Why can other implementations claim compliance when we can't even get a chance to verify it?
Really does say a lot about the corporate-driven JCP and politics. The JCP is inefficient, secretive, and doesn't present a level playing field for all.

PS1. When I tried to get the JPA1 TCK back in 2007 it took 3 months of continual emailing before I finally got my hands on it. That was when they weren't merging with anyone else, so seems to be fairly typical of the wasteful process they have.

PS2. JDO obviously has an open TCK, Apache2 licensed, available for all to download and contribute to. You don't need to sign NDA's just to see it.

Update (22nd June 2010) : Now more than 4 and a half months have passed since the original request for the JPA2 TCK. Still nothing.

So do you think Oracle is deliberately delaying this ? or is it just inefficient and disorganised ? Answers on a postcard please. The point that is becoming clearer is that JPA "compliance" is not worth the effort and DataNucleus releases will go forward claiming compliance since the JCP is preventing us from participating in this little game of theirs.andyhttp://firstname.lastname@example.com:blogger.com,1999:blog-1936942592343397099.post-56992336475654746652010-03-04T02:02:00.000-08:002010-03-05T09:47:35.680-08:00JDOQL2 statusSome time ago I started a new implementation of JDOQL for RDBMS, to replace the legacy (TJDO) JDOQL implementation. The legacy implementation had problems with generation of SQL for many queries, particularly involving "contains", "!contains" and use of unbound variables. As described here the replacement implementation builds on the generic query compilation used for all other datastores.

With the release of AccessPlatform 2.0.2 "JDOQL2" is ready for use in the vast majority of situations. It fixes various issues with the legacy implementation, successfully generating SQL for many queries that failed with legacy. It still doesn't compile every complicated query perfectly, but it is at least understood what parts it is failing on now, and none of those parts in general work in the legacy implementation.

In AccessPlatform 2.1 "JDOQL2" will be renamed to "JDOQL", and the legacy implementation will become "JDOQL-Legacy" so you will get this new implementation by default, and have to explicitly select the legacy implementation if you wish to use it.

By AccessPlatform 2.2 we intend to remove the legacy JDOQL and just move forwards with this new implementation.andyhttp://firstname.lastname@example.com:blogger.com,1999:blog-1936942592343397099.post-56783717567394026892010-01-28T00:08:00.000-08:002010-06-08T08:04:24.921-07:00GAE/J and JDO/JPAIn April 2009 Google announced AppEngine for Java, providing support for JDO and JPA. Since that time people have had the chance to see what they think of it and identify shortcomings. This has lead to many misconceptions of JDO and JPA (though mainly JDO) due to thinking that what Google provide is a full and true representation of these standards; it isn't. This blog entry attempts to correct some of these misconceptions, and to suggest some areas where Google could remedy the situation. All of the following items would help in providing a true reflection of these persistence standards and aid GAE/J users in having (much more) portable applications.

JDO/JPA : Lack of support for many methods and operators in JDOQL/JPQL

GAE/J only supports a relatively small subset of the available methods and operators of JDOQL/JPQL. This is due to the underlying datastore not supporting certain capabilities in its queryability. The problem is that this gives the impression that JDOQL/JPQL are somehow weak. What would make way more sense would be to evaluate all that is evaluatable in the datastore, and then have a flag set while compiling for whether the query contains any feature that the datastore query cannot handle and, if so, run the DataNucleus in-memory query evaluator on the resultant instances. This would provide a transparent interface to JDOQL/JPQL and mean that people don't have to have build some custom queries just to get around GAE/J shortcomings.

JDO : No support for input candidate collection for JDOQL

GAE/J does not support inputting a candidate collection and query over that collection. This is a trivial thing to support, particular since the code necessary to do it was contributed some time ago, yet isn't in the current plugin.

JDO : pm.getExtent doesn't handle subclasses

The current pm.getExtent() implementation in GAE/J doesn't support the subclasses flag. The root cause is that the underlying datastore doesn't support a single query to retrieve a class and its subclasses. The simple solution would be to run "n" queries on the datastore, one for each of the possible subclass types, and merge the results. This would be simple to do and implement, and would provide correct JDO behaviour so users don't see any shortcoming.

JDO/JPA : exposing Google-specific id classes

With GAE/J there is some flexibility on what is allowable as a PK field; Long, String, and "Key". The first two are standard classes and standard JDO. The latter is environment specific. Firstly GAE/J ought to allow short/int/Short/Integer/long for true JDO/JPA operation so that users see no difference. Secondly, the "id" exposed to the user should be a JDO or JPA id and should be portable. When we implemented support for persisting to db4o we didn't expose db4o's id, instead wrapping it internally, so the user has no unexpected classes popping up that prevent their migration elsewhere later on.

JDO/JPA : one entity group per transaction

With GAE/J you have a restriction on the number of "entity groups" that can be enlisted in any transaction. Ok, but why expose this to the user and restrict what they do ? The logical way to do it would be to have multiple "internal" transactions for a JDO/JPA transaction and have each of these for a particular entity group. Since the underlying datastore doesn't provide ACID transactions anyway there is little impact of doing this. It would then mean that you don't impose on users having to split their persistence code apart just to get it to run, and hence mean that it is portable

Unowned relations

This term is being used in GAE/J seemingly where you have a Collection<id> and so no real relation, although the "id" relate to other objects. This is perfectly representable in JDO/JPA as a Collection<long>, or Collection<String> or even Collection<Object>. With one of these "relations" the onus is on the user to manage the relation.

Support for types persistable as String

DataNucleus has, for some time, provided a mechanism for defining how to persist a type as a String (and retrieve its value from the String) - see ObjectStringConverter in DataNucleus "core" code. GAE/J could easily provide support for this in their plugin (if a type is not natively supported then check if there is an ObjectStringConverter and use that) and this would mean that many more Java types are persistable using AppEngine.

Documentation : @Persistent

In the GAE/J docs, every field has @Persistent marked against it. This is totally unnecessary, and you only need @Persistent for a non-standard field type. It leads to people believing that you must specify this to get something persisted, and so when they want to have a field not persisted they just remove the annotation. Please update the docs to reflect the minimal configuration required so we give a fair reflection of JDO and its spec. For example

@PersistenceCapable
public class MyClass
{
 @PrimaryKey
 Long id;

 String name;

 double value;

 ...
}

Package naming "org.datanucleus.*"

This plugin is provided by Google not DataNucleus. It's currently packaged as "org.datanucleus.store.appengine". This leads to people believing that DataNucleus itself is at fault for its shortcomings. This is unacceptable and we own the domains datanucleus.org/datanucleus.com. Please rename your packages ASAP.

Nowhere have we seen any attribute of the GAE/J BigTable datastore that cannot be handled by the JDO or JPA API's. The JDO API (and metadata) in particular was designed as generic, and there is nothing in a "NoSQL" datastore that should cause it any problems with representation. We challenge anyone to define where there is such a problem area and it can then be addressed (there's a JIRA open on the Apache JDO project for just this situation); if you really can come up with a problem area then its in all of our interests to understand it and tackle it.andyhttp://firstname.lastname@example.com


text:
I miss blogging
Walk as if you are kissing the Earth with your feet.
—Thích Nhất Hạnh (via paintedthemtiedye)
(Source: purplebuddhaproject, via gnostic-forest)
I dont deserve this, I tried so hard.
What a catch!
Photo by Roman Golubenko
It feels so good to be doing something I love and knowing that if I follow this path, I'll have a satisfying, significant life.
Updates on a killer fucking week
Aka, a rant about all the good shit in my life
Appreciating more and more the friendship that will always be there with my sisters, no matter what happens.
(Source: givemeallthebaconandeggs, via awelltraveledwoman)
Lone tree, Hartrigg, Kentmere, Lake District National Park, England by Ministry
(Source: wanderthewood, via awelltraveledwoman)

subset: commoncrawl


text:
this wiki
Lugdunum (modern: Lyon, France) was a city located in the Roman Empire, and also home to the Assassin Brotherhood, including the Assassins Aquilus and his father Lucius.
After his confrontation with general Gracchus, Aquilus traveled back to his hometown to bring the Ankh to his father.[1]
When he arrived in Lugdunum, he found his father in good health, and met a family friend, Senator Caius Fulvus Vultur. Aquilus almost immediately set out to find the person responsible for his betrayal, and so he sought out Faustin. After a brief interrogation, Faustin betrayed his accomplice: Senator Vultur.[1]
When Aquilus arrived at his house once again, he found his father dead and the Ankh stolen. One of the maids told him the senator had fled the city.[1]
Not much later, the Alamans moved in on Lugdunum, though their leader, Accipiter, was hesitant to plunder the city because of the Assassins' interests. Instead of plundering it, a deal was made. The city paid a large sum of gold to keep the Alamans from invading the city, which made an easy victory for the Alamans.[2]
- ↑ 1.0 1.1 1.2 Assassin's Creed (French comic): Tome 2, Aquilus
- ↑ Assassin's Creed (French comic): Tome 3, Accipiter


text:
A young and mischievous witch named Fuchsia enrolls in witch school to learn magic. Against her father's warning, Fuchsia befriends a human boy named Tommy and learns about his uncle's plans to destroy her school. It is up to Fuchsia and her new magical powers to save her town!
Rated PG by the Motion Picture Association of America for some mild rude humor and smoking.


text:
I honestly don't get Kim Kardashian's body. It literally defies all science and logic. I mean, I saw the world-renowned KUWTK episode where she got an X-ray of her behind, proving it's au naturel, butt ... how is that possible? This new picture of Kim Kardashian's butt is mind-boggling. Literally! Until we saw it on Us Weekly, my co-workers and I were debating about whether it was fake or not.
Kim has really made an effort to lose the baby weight since becoming a mom, and it seems like all of her hard work has paid off. Her tummy is back to being flat, her waist trim, but ... I just don't get how her famous backside manages to look the same. Tell us your secrets, Kim! This is one of the world's great wonders!
Do you think Kim's had any enhancements to her, ah, butt?
Image via Kim Kardashian/Instagram
Wow! How do you know that Kim, Court, and Chloe removes fat from their tmmy and inject into their hips? And does that make them bad people? I get it that people like making fun of them because they are famous for nothing, but I happen to like them because they are famous for nothing. Leave them alone and let them have their last season of The Kardashians in peace.


text:
A Seabrook man has been arrested in connection with two armed robberies of Burton-area businesses.
The Beaufort County Sheriff's Office says Marvin Ellis Frazier, 46, was taken into custody at his Delaney Circle home.
Officials say witnesses were able to identify Frazier as the suspect in the January 2 armed robbery of the China Wok restaurant on Shanklin Rd.
Investigators also believe he is responsible for the January 5 robbery of the Sunhouse Petroleum convenience store on Trask Parkway.
Frazier is charged with Armed Robbery, Kidnapping and other crimes.
He is a convicted felon, authorities said.
1430 East Victory DriveSavannah, GA 31404
Telephone: 912.651.0300
Fax: 912.651.0320
Email: firstname.lastname@example.com