During a recent upgrade, we encountered one particularly difficult and challenging problem to solve.
This application used Hibernate Envers for auditing, and Liquibase via the Grails dbmigrations plugin for change management of the database schema.
The upgrade we were performing was a significant upgrade, with thousands of classes, and hundreds of data domains however we had completed the majority of the work and were now testing final tasks.
One incorrect assumption on our behalf was that we wouldn’t face any issues with the ‘dbmigration‘ plugin since we’d already tested it early on, but only to ensure it ran correctly and would run the status task correctly. How wrong we were.
Despite using the same database in our new Grails 5 version as in the original Grails 2 version, whenever we ran a “gorm diff” task to identify any changes, eventually after a significant time, the result was that it was falsely detecting thousands and thousands of diffs.
After inspection of the changelogs, I could see the majority of the detections were related to column defaults, specifically how default NULL values were represented.
By adding defaultValue: ‘NULL’ to the domain ‘mapping’ closures, we were able to reduce the detected changesets from over 900’000, to around 300’000. A big drop but still a lot of work to do.
The majority of remaining changesets were now related to Boolean fields and the default NULL problems. There did not seem to be a working mapping configuration for NULL Booleans that did not trigger detected schema changes until I discovered if we specified a defaultValue: ‘false’ then it stopped creating the ‘dropDefaultValue’ changesets. It didn’t make sense, but it let us move forwards.
Once that was addressed, we significantly reduced the detected changesets from 300’000 to only about 30, and of these remaining change detections, all were related to the tables that the Envers auditing library creates.
These appeared to be clear bugs within hibernate Envers and the way it was detecting foreign keys. and also that all these remaining tables were tables that extended abstract classes.
We also noticed that if we allowed these changesets to run and update the DB, then the next diff would result in exactly the same changes detected, it was a repeating scenario.
We spent some time tracing through the internals of Hibernate, and Liquibase in an attempt to find the root cause and if we could fix this issue, and even reached out to the Grails team at OCI as we felt that surely, they must have experienced the same issue, assuming they’ve assisted their customers in upgrading system which used the combination of Envers and Liquibase. Unfortunately not, we were on our own but had to find a solution.
A Eureka moment was reached when I noticed that there was a discrepancy in the case of the column names Envers was attempting to create. In our existing tables, ‘_rev’ and ‘_revtype’ columns were lowercase whilst in the suggested changesets Liquibase was attempting to create ‘_REV’ and ‘_REVTYPE’ columns. A similar issue was due to upper and lowercase _AUD / _aud table names.
It appeared the Envers system would not initialise in the correct order, when calling the dbmigration tasks, causing it to fail to recognise the Liquibase generated tables correctly.
Our final solution to solve these remaining problems, was to resort to using Java reflection and override core Hibernate functionality. Something I really wished we didn’t have to do, but so far we had no solution offered from the Grails team, no solution offered from the Hibernate community, and no solution from Liquibase. Providing the following code, allowed us to ensure the audit tables were correctly created, with lowercase field names which then prevented the recurring change detections.
@Configuration
class CustomEnversConfig {
private static final String ENVERS_AUDIT_TABLE_SUFFIX = 'org.hibernate.envers.audit_table_suffix'
private static final String ENVERS_REVISION_FIELD_NAME = 'org.hibernate.envers.revision_field_name'
private static final String ENVERS_REVISION_TYPE_FIELD_NAME = 'org.hibernate.envers.revision_type_field_name'
static {
modifyFinalField(EnversServiceInitiator, "INSTANCE", new CustomEnversServiceImplInitiator())
}
private static void modifyFinalField(Class<?> targetClass, String fieldName, Object newValue) {
Field targetField = ReflectionUtils.findField(targetClass, fieldName)
int modifiers = targetField.modifiers
Field modifierField = Field.getDeclaredField("modifiers")
modifierField.accessible = true
modifierField.setInt(targetField, modifiers & ~Modifier.FINAL)
ReflectionUtils.setField(targetField, null, newValue)
}
static class CustomEnversServiceImplInitiator extends EnversServiceInitiator {
@Override
EnversService initiateService(Map configurationValues, ServiceRegistryImplementor registry) {
configurationValues[ENVERS_AUDIT_TABLE_SUFFIX] = '_aud'
configurationValues[ENVERS_REVISION_FIELD_NAME] = 'rev'
configurationValues[ENVERS_REVISION_TYPE_FIELD_NAME] = 'revtype'
return new CustomEnversServiceImpl()
}
}
static class CustomEnversServiceImpl extends EnversServiceImpl {
@Override
void initialize(final MetadataImplementor metadata, final MappingCollector mappingCollector) {
Field initializedField = ReflectionUtils.findField(this.class, "initialized")
ReflectionUtils.makeAccessible(initializedField)
initializedField[this] = false
super.initialize(metadata, mappingCollector)
}
}
}If you have found this article as a result of searching for a solution to the same problem, then I hope this will save you the time and pain we endured. It is not often we encounter challenges quite like this, but as with most software development problems, it serves as a reminder that no matter how difficult finding the solution might be, or how obscure the problem might be, if we don’t stop and keep digging, eventually, we WILL find a solution.
