Friday, March 28, 2008

SimpleSync: a synchronizer for ANYTHING

 
Step One: Set up a few models -- use SimpleMapper, DataMapper, ActiveRecord, ActiveResource, my Quickbooks gem, or any other ORM you can get to respond correctly to the few essential methods: obj = Model.get(finder_options={}); obj = Model.new(attributes={}); obj.save; obj.delete.

Step Two: Follow the five simple steps to synchronize your data source:

1) Create a syncer object

@sync = SimpleSync::Syncer.new( <last_sync_time> )
2) Add some sources
@bug_source = @sync.add_source(<source_model>, <finder_options>, <common_id_attribute_name (example: id)>)
@juice_source = @sync.add_source(<source_model>, <finder_options>, <common_id_attribute_name (example: bug_id)>)
3) Define for each source how to grab updated records to sync
@bug_source.new_records {|start_time| <block code that returns record objects> }
@bug_source.changed_records {|start_time| <code that returns record objects> }
@bug_source.deleted_records {|start_time| <code that returns record objects> }
@juice_source.new_records {|start_time| <code that returns record objects> }
@juice_source.changed_records {|start_time| <code that returns record objects> }
@juice_source.deleted_records {|start_time| <code that returns record objects> }
4) Map the attribute-conversions for all attributes you want to synchronize
@sync.mappings[@bug_source => @juice_source] = {
'first_name' => 'FirstName',
'last_name' => 'LastName',
'company_name' => 'JuiceCompanyName',
'birthdate' => 'Birthday',
}
# And usually just an inversion of the first hash, but sometimes not:
@sync.mappings[@juice_source => @bug_source] = @sync.mappings[@bug_source => @juice_source].invert
5) Sync!!
@sync.sync!
6) Wait for a while, then sync again!
@sync.sync!

The purpose of SimpleSync is to handle all of the logic of synchronizing and leaving all the specifics of the data to your custom models and attribute mappings. The above code was taken from a working script and slightly modified just to better illustrate the purposes of each piece.

Install the gem:
gem install simplesync

Now get this: I have a fully-functional synchronizer between Quickbooks Customers and my addressbook website (not yet public), using 3 of my gems: SimpleSync, SimpleMapper, and Quickbooks -- in a script of only 150 lines. Beat that, I dare you. :)
 

3 comments:

crayz said...

This sounds really awesome, I'll check it out. Have you used this with any large datasets?

daniel said...

No, so far this is like a first-level synchronizer, as it has only one specific method of synching -- it doesn't compare records, it doesn't compare updated_at times or anything, it just asks for a list of new, changed, and deleted records from each source and loops through the source copying and saving the records appropriately.

The other thing is, for large data-sets, I haven't put in any type of paginating mechanism .. so when a large number of items need synced, it'll use a lot of memory.

Perhaps in a later version I can add these options, and of course your data sources (models) must support them as well.

Also, I may possibly put up a page where different data-source plugins can be provided, specifying what attributes and options are available .. that way you could just include a random sample of two sources and voila! Does this remind you of Apple's iSync, by chance?

Roger Pack said...

http://code.google.com/p/ruby-roger-useful-functions/wiki/TableSyncer does straight mysql table syncs, as a note :)