Easing multi-module development with Ivy
I am developing a build system and Ivy is an important part of it. One thing I need to support is easy development on multiple modules of a project simultaneously. Changes to one module need to be picked up by other modules. This has to be very easy to use because users have to do this a lot of times.
After browsing and searching through the forums and the site I found out that more people were struggling with the same problem. Below is my proposal that will allow easy development on multiple modules with a minimal impact on Ivy.
Use case
Users work on many modules of a project at the same time. When they make changes to a project, they want these changes to be picked up in the Ant builds of other projects, and in their IDEs.
Requirements
The following requirements are based on the assumption that users are doing this a lot of times during their daily work. Therefore the requirements focus on minimal overhead for the user.
Users should not have to touch files under version control. If a file under version control needs to be changed it will contain development information which could be committed by accident.
The users should only have to change a single item to switch to the development revision.
The recreation of the class path of a project in an IDE should only be necessary when either the dependencies or the revision numbers of the dependencies of the project change.
Ivy does not have to set up the complete environment, because there are too many different development environments. Ivy only needs to provide the information so it can be done by the user.
Solution
Switching to the development revision
To change the revsion of a dependency to the development revision, the user only needs to change a single property containing the revision of the dependency. This is done in a properties file which is not under version control. The properties file overrides another properties file containing the stable revisions of the dependencies. The latter properties file is under version control. Both files have to be included in the Ant build file. For example:
File local-dependencies.properties (not versioned, overrides dependencies.properties):
foo.rev=1.02-devFile dependencies.properties (versioned):
foo.rev=1.01File ivy.xml:
...
<dependency org="foo-org" name="foo" rev="${foo.rev}">
...The above is already possible with the current version of Ivy.
Picking up changes to artifacts
Currently the file system resolver only detects changes to Ivy files when checkmodified is set to true. The file system resolver can be extended so it will also check other artifact types. By adding attribute checktypes the user can specify what needs to be checked in his Ivy configuration. The attribute accepts either ivy or all, with ivy being the default value for backwards compatibility. For example:
File ivyconf.xml or one of its referenced files:
...
<filesystem name="local" checkmodified="true" checktypes="all">
...Updating IDE class paths
The class paths of IDE projects consist of two parts: the file specifying the class path and the retrieved artifacts. The retrieved artifacts will be updated during an Ant build. Doing an Ant build to update the class path of IDE projects is undesirable, because an updated artifact can be used by many projects.
If Ivy provides information about the repository location of an artifact, this information can be used to let the class paths of IDE projects point directly to the local repository. By generating information similar to what is shown below the user can use XSLT to generate the classpath for his IDE:
<modules>
<module org="foo-org" name="foo" rev="1.02-dev" status="integration">
<artifact name="foo-unoptimized" type="jar" ext="jar">
<retrieval-location>C:/prj/bar/lib/compile/foo-unoptimized-1.02-dev.jar</retrieval-location>
<local-location resolver="local">C:/data/ivy/local/foo-org/foo/1.02-dev/foo-unoptimized-1.02-dev.jar</local-location>
</artifact>
<artifact name="foo-src" type="zip" ext="zip">
<retrieval-location>C:/prj/bar/lib/compile/foo-src-1.02-dev.zip</retrieval-location>
<local-location resolver="local">C:/data/ivy/local/foo-org/foo/1.02-dev/foo-src-1.02-dev.zip</local-location>
</artifact>
</module>
<module org="junit" name="junit" rev="3.8.2" status="release">
<artifact name="junit" type="jar" ext="jar">
<retrieval-location>C:/prj/bar/lib/compile/junit-3.8.2.jar</retrieval-location>
<remote-location resolver="default">http://www.ibiblio.org/maven/junit/jars/junit-3.8.2.jar</remote-location>
</artifact>
</module>
</modules>By using the location in the local repository of the foo artifacts, the IDE will detect changes as soon as foo is published to the local repository.
The information should probably be generated by a separate Ant task as it contains information known only to resolve or only to retrieve, so generating it as a side effect of one of these two tasks seems unnatural.
It must be possible to specify the string to use as a file separator, so users don't have to convert paths in XSLT.
Notes
Testing newer dependencies
The setup described above is also useful for testing whether a project still behaves correctly with new revisions of modules, without having to touch files under version control. The user only needs to make changes to local-dependencies.properties. For example, to test if everything still works with JUnit 3.8.2 and jMock 1.0.1:
File local-dependencies.properties:
junit.rev=3.8.2
jmock.rev=1.0.1File dependencies.properties:
junit.rev=3.8.1
jmock.rev=1.0Paths and filenames
The paths and filenames used throughout the examples are completely customizable to suit the needs of a specific environment.
Conclusion
The proposed changes above will make it a lot easier to work with Ivy during development. The impact on Ivy is, as far as I can tell, minimal and designed to be backwards compatible.
I would love to hear what you, Ivy developers and users, think about it.
Johan Stuyts
The changes to Ivy are:the
The changes to Ivy are:
- the
checktypesattribute - and an Ant task generating an XML file similar to the one I posted.
I think that linking a revision to a build is a great idea for published modules because users have to be able to determine whether or not they use exactly the same build. But local modules are not shared, making me the only user and I don't need to be able to identify each of the builds.
I am very interested in your tutorial and will try it out to see if it works for me. The only issue I have with your process is that users do not have to explicitly state that a development revision should be used for a dependency. By making this explicit, by having to set a property for example, I feel users get a better idea of how things are structured and working. Also, it is impossible to accidentally pick up a local version if a user forgets to call clean-local.
I have not looked at IvyIDE yet. I will give it a try soon. The reason I want to generate the class path of my IDE (Eclipse) is, that I want the build system I am developing to work out of the box without users having to install a multitude of tools and plugins. To be able to generate the class path I need the information in the XML.
--
Johan Stuyts
For the ant task, I'm not
For the ant task, I'm not sure it should really be part of Ivy... but it doesn't matter much if it is part of ivy official distrib or not. I f someone develop it and if several uses it, we will add it to standard distrib (if the one who developed it is ok, for sure).
Now conerning module identification, I personnaly think it's interesting even with local modules, because I need to run a resolve in my project to benefit of the latest local version I just published, and in this case it's interesting to know its version. But I admit it doesn't make a real advantage over your solution.
Concerning the setting of a properties file over calling a task, I don't know which is better, it's mainly a matter of preference... What I like with our solution is that when I want to use the local version of a module I only have to do something in the concerned module (call publish-local) and then all modules will use the local version (we call resolve very frequently). With your solution I have to modify the property file in each modules... except if the properties file is unique for a developer, which is maybe what you advise but I didn't understand it like that.
Finally, concerning IvyDE, if you use eclipse I strongly encourage you to give it a try. Installing a plugin is not very long, and the benefit of using a classpath container is important: I can't detail why, but using it in our day to day job makes us realize the time it saves... but once again maybe it's only a matter of preference.
__________________________________
Xavier Hanin
Jayasoft team member
I will have a look at the
I will have a look at the source to see if I can implement the Ant task I described.
I have not had the need to be able to identify different builds of local releases. We use Maven at the moment and when I rebuild a project and install it locally using the same version number, the other projects and the class path of Eclipse automatically pick up the changes. This is the simplicity I want to have with Ivy too.
There is indeed a bit more overhead with my way of using development revisions. Each project has its own file for specifying development properties. But this allows the user to specify per project whether or not he wants a development or an official version of a dependency.
I will not avoid using IvyIDE (or other tools and plugins for that matter), but I want to have an out-of-the-box solution too. Also, I want to be able to support users which do not use Eclipse.
--
Johan Stuyts
Allright, I think I now
Allright, I think I now better see how you would like things to go, and I can understand you prefer your way. Maybe what could be more compatible with my point of view would be to be able to use a special revision suffix (dev ?) which would automatically set the changing attribute to true. Do you see what I mean ? With this behaviour you don't even have to set anything else than the dependency revision, and one can now just by looking at the ivy file that the revision is changing (if he knows the meaning of the special revision). I think this may match both your usage and my preference to keep this info in the ivy file. What do you think ?
Oh, and concerning IvyDE, I clearly understand that you want to support non eclipse users and out of the box solution, it's indeed a very good thing to be able to address the most possible number of users.
__________________________________
Xavier Hanin
Jayasoft team member
Your solution sounds very
Your solution sounds very good to me too. If you make it some kind of regular expression pattern that can be configured, then I think a lot of users will be helped. So skip the checktypes attribute and implement what you suggest.
I will still implement the Ant task to generate information about the location of artifacts, so I can create the class path using XSLT.
--
Johan Stuyts
I don't know if I'll be able
I don't know if I'll be able to implement what I suggest soon, but I think it's worth adding a feature request in jira 
Then we will have to see which features get priority over others, some are waiting for other issues to be solved for a long time now. But if you submit a patch with the issue, there are good chance the feature will be integrated in ivy sooner
__________________________________
Xavier Hanin
Jayasoft team member
I have implemented the
I have implemented the system I described and have made the following changes to Ivy.
Local changes with same revision
For the detection of changed artifacts in the repository I added attribute changingPattern to AbstractResolver. BasicResolver will use this pattern, in addition to the changing value in the Ivy file, to determine whether the artifacts of a module are changing.
Because Ant does not detect changes to JAR files, Java sources were not recompiled after a retrieve which actually copied targets. I changed Ivy to return the number of copied targets of a retrieve, and I changed IvyRetrieve to set property ivy.targets.copied to true if the number of targets is greater than zero (and to false otherwise). Build scripts can use this to delete compiled classes if needed.
Artifact report
To be able to generate a report about the artifacts I had to let BasicResolver store the origin of the artifact, and whether or not the origin is local, next to the artifact in the cache. This is stored in a properties file called {path in cache of the artifact).origin.
The interface Resource has been extended with a method so BasicResolver can determine whether or not a resource is local.
I have written a new Ant task to combine the resolved dependencies, the artifacts that retrieve will copy, and the origins of the artifacts in the cache, to an artifact report in XML. I had to refactor Ivy so I could get the artifacts that would be copied without actually copying them. Per module, the report contains:
- The organisation, name, revision and status
- Information about all the artifacts of the module
Per artifact, the report contains:
- The artifact name, extension and type
- The origin of the artifact, and whether or not is is local
- The location in the cache of the artifact
- All locations to which
retrieve will copy the artifact
Using the report I can now generate an Eclipse class path which points development revisions directly to the local repository. It also attaches a sourcepath if it can find an artifact for the source (which will also point to the local repository for development revisions). After publication of another project I only have to refresh the project to have Eclipse recompile it.
Publication issues
I think there is an issue concerning publication. If you have an existing delivered Ivy file, this file will get copied to the local repository instead of an Ivy file with the publication date that you specify. I fixed this by deleting the artifacts before a publication in my build script.
Next steps
I will create separate patches for local changes with same revision and artifact report in the next couple of days. I will submit these patches to JIRA.
--
Johan Stuyts
Nice, you were so quick
Nice, you were so quick !
I'll have a look to your patch when you'll submit them, and will do my best to integrate them in a future official version of Ivy. Thanks a lot for your contribution.
__________________________________
Xavier Hanin
Jayasoft team member
Configuration example
Do you have a configuration example for the solution which is implemented with the patch? I'm not sure howto configure the changingPattern for a module.
Example config for 'changingPattern'
Hi Marco,
Here it is. You just have to add attribute changingPattern to a resolver. The format is a regular expression. Any revision matching the regular expression will become a changing revision. In the example below it is any revision ending with -dev:
<filesystem name="local" changingPattern="^.+-dev$">
<ivy pattern="${ivy.local.default.root}/${ivy.local.default.ivy.pattern}"/>
<artifact pattern="${ivy.local.default.root}/${ivy.local.default.artifact.pattern}"/>
</filesystem>--
Johan Stuyts
Thanks for sharing your
Thanks for sharing your point of view, Johan, we really appreciate it.
First, I'm not sure to have understood exactly what is needed in Ivy... A checktypes attribute on the resolver to tell it if it should check if the jars have changed, and that's all ?
Note that I'm not very pleased with the idea, but it's only my opinion. In ivy philosophy a revision identifies a specific build. That's why when this is not the case, I prefer to be able to see it in the ivy file (by setting changing to true). In your case it's only the ivyconf which changes this behaviour. But I understand the need to ease multi modules development.
In my company we also do a lot of multi-module development, and we use ivy in another way... When a developer want to use his own revision of a module he doesn't have to change a properties file, he just call a task called publish-local in the module for which he wants to use his own revision. Then he calls a resolve in the modules where he want to benefit from this revision, and that's all. The revision, called a local revision, is put in a local repository at the beginning of a chain with a return first set to true (it is now the default configuration). When he updates the revision, he does the same thing (publish-local + resolve) When he doesn't want to use this revision anymore, he calls a task called clean-local, and resolve in the other project. We have a tutorial in progress about that, we should definitely take the time to finish it. And for IDE point, we use IvyDE and it works like a charm.
What we like in this solution is that we can always know which revision of a dependency is used: a local or an official one, and in each case which local or official one (was it the one from 1 hour or 2 hours ago I used when I ran this test ?).
But I would really pleased to hear from other users what do they think about your solution. We do not pretend to own the only truth about software development, so if several of you like Johan approach, we could better see what could be done in future versions of ivy.
__________________________________
Xavier Hanin
Jayasoft team member