class GenericFormatter < Formatter attr_accessor :tag_name,:objects def generate_xml builder = Nokogiri::XML::Builder.new do |xml| xml.send(tag_name.pluralize) { objects.each do |obj| xml.send(tag_name.singularize){ self.generate_obj_row obj,xml } end } end builder.to_xml end def initialize tag_name,objects self.tag_name = tag_name self.objects = objects end def generate_obj_row obj,xml obj.attributes.except("updated_at").map do |key,value| xml.send(key, value) end xml.updated_at obj.updated_at.try(:strftime,"%m/%d/%Y %H:%M:%S") if obj.attributes.key?('updated_at') end end In the above code, I have implemented the formatter where I have used the nokogiri XML Builder to generate the XML by manipulating the objects passing out inside the code.It's generated the faster XML when the data is not too large if data is larger like more than 10,000 records then It's slow down the XML to generate and takes at least 50-60 seconds.
Problem: Is there any way to generate the XML faster, I have tried XML Builders on view as well but did n't work.How can I generate the XML Faster? Should the solution be an application on rails 3 and suggestions to optimized above code?
2 Answers
Answers 1
Your main problem is processing everything in one go instead of splitting your data into batches. It all requires a lot of memory, first to build all those ActiveRecord models and then to build memory representation of the whole xml document. Meta-programming is also quite expensive (I mean those send methods).
Take a look at this code:
class XmlGenerator attr_accessor :tag_name, :ar_relation def initialize(tag_name, ar_relation) @ar_relation = ar_relation @tag_name = tag_name end def generate_xml singular_tag_name = tag_name.singularize plural_tag_name = tag_name.pluralize xml = "" xml << "<#{plural_tag_name}>" ar_relation.find_in_batches(batch_size: 1000) do |batch| batch.each do |obj| xml << "<#{singular_tag_name}>" obj.attributes.except("updated_at").each do |key, value| xml << "<#{key}>#{value}</#{key}>" end if obj.attributes.key?("updated_at") xml << "<updated_at>#{obj.updated_at.strftime('%m/%d/%Y %H:%M:%S')}</updated_at>" end xml << "</#{singular_tag_name}>" end end xml << "</#{tag_name.pluralize}>" xml end end # example usage XmlGenerator.new("user", User.where("age < 21")).generate_xml Major improvements are:
- fetching data from database in batches, you need to pass ActiveRecord collection instead of array of ActiveRecord models
- generating xml by constructing strings, this has a risk of producing invalid xml, but it is much faster than using builder
I tested it on over 60k records. It took around 40 seconds to generate such xml document.
There is much more that can be done to improve this even further, but it all depends on your application.
Here are some ideas:
- do not use ActiveRecord to fetch data, instead use lighter library or plain database driver
- fetch only data that you need
- tweak batch size
- write generated xml directly to a file (if that is your use case) to save memory
Answers 2
The Nokogiri gem has a nice interface for creating XML from scratch, Nokogiri is a wrapper around libxml2.
Gemfile gem 'nokogiri' To generate xml simple use the Nokogiri XML Builder like this
xml = Nokogiri::XML::Builder.new { |xml| xml.body do xml.test1 "some string" xml.test2 890 xml.test3 do xml.test3_1 "some string" end xml.test4 "with attributes", :attribute => "some attribute" xml.closing end }.to_xml output
<?xml version="1.0"?> <body> <test1>some string</test1> <test2>890</test2> <test3> <test3_1>some string</test3_1> </test3> <test4 attribute="some attribute">with attributes</test4> <closing/> </body> 
0 comments:
Post a Comment