Showing posts with label nokogiri. Show all posts
Showing posts with label nokogiri. Show all posts

Wednesday, May 23, 2018

Parse 'ul' and 'ol' tags

Leave a Comment

I have to handle deep nesting of ul, ol, and li tags. I need to give the same view as we are giving in the browser. I want to achieve the following example in a pdf file:

 text = " <body>     <ol>         <li>One</li>         <li>Two              <ol>                 <li>Inner One</li>                 <li>inner Two                      <ul>                         <li>hey                              <ol>                                 <li>hiiiiiiiii</li>                                 <li>why</li>                                 <li>hiiiiiiiii</li>                             </ol>                         </li>                         <li>aniket </li>                     </li>                 </ul>                 <li>sup </li>                 <li>there </li>             </ol>             <li>hey </li>             <li>Three</li>         </li>     </ol>     <ol>         <li>Introduction</li>         <ol>             <li>Introduction</li>         </ol>         <li>Description</li>         <li>Observation</li>         <li>Results</li>         <li>Summary</li>     </ol>     <ul>         <li>Introduction</li>         <li>Description              <ul>                 <li>Observation                      <ul>                         <li>Results                              <ul>                                 <li>Summary</li>                             </ul>                         </li>                     </ul>                 </li>             </ul>         </li>         <li>Overview</li>     </ul> </body>" 

I have to use prawn for my task. But prawn doesn't support HTML tags. So, I came up with a solution using nokogiri:. I am parsing and later removing the tags with gsub. The below solution I have written for a part of the above content but the problem is ul and ol can vary.

     RULES = {   ol: {     1 => ->(index) { "#{index + 1}. " },     2 => ->(index) { "#{}" },     3 => ->(index) { "#{}" },     4 => ->(index) { "#{}" }   },   ul: {     1 => ->(_) { "\u2022 " },     2 => ->(_) { "" },     3 => ->(_) { "" },     4 => ->(_) { "" },   } }  def ol_rule(group, deepness: 1)   group.search('> li').each_with_index do |item, i|     prefix = RULES[:ol][deepness].call(i)     item.prepend_child(prefix)     descend(item, deepness + 1)   end end  def ul_rule(group, deepness: 1)   group.search('> li').each_with_index do |item, i|     prefix = RULES[:ul][deepness].call(i)     item.prepend_child(prefix)     descend(item, deepness + 1)   end end  def descend(item, deepness)   item.search('> ol').each do |ol|     ol_rule(ol, deepness: deepness)   end   item.search('> ul').each do |ul|     ul_rule(ul, deepness: deepness)   end end  doc = Nokogiri::HTML.fragment(text)  doc.search('ol').each do |group|   ol_rule(group, deepness: 1) end  doc.search('ul').each do |group|   ul_rule(group, deepness: 1) end     puts doc.inner_text   1. One 2. Two  1. Inner One 2. inner Two  • hey  1. hiiiiiiiii 2. why 3. hiiiiiiiii   • aniket    3. sup  4. there   3. hey  4. Three    1. Introduction  1. Introduction  2. Description 3. Observation 4. Results 5. Summary    • Introduction • Description  • Observation  • Results  • Summary       • Overview 

Problem

1) What I want to achieve is how to handle space when working with ul and ol tags
2) How to handle deep nesting when li come inside ul or li come inside ol

2 Answers

Answers 1

I've come up with a solution that handles multiple identations with configurable numeration rules per level:

require 'nokogiri' ROMANS = %w[i ii iii iv v vi vii viii ix]  RULES = {   ol: {     1 => ->(index) { "#{index + 1}. " },     2 => ->(index) { "#{('a'..'z').to_a[index]}. " },     3 => ->(index) { "#{ROMANS.to_a[index]}. " },     4 => ->(index) { "#{ROMANS.to_a[index].upcase}. " }   },   ul: {     1 => ->(_) { "\u2022 " },     2 => ->(_) { "\u25E6 " },     3 => ->(_) { "* " },     4 => ->(_) { "- " },   } }  def ol_rule(group, deepness: 1)   group.search('> li').each_with_index do |item, i|     prefix = RULES[:ol][deepness].call(i)     item.prepend_child(prefix)     descend(item, deepness + 1)   end end  def ul_rule(group, deepness: 1)   group.search('> li').each_with_index do |item, i|     prefix = RULES[:ul][deepness].call(i)     item.prepend_child(prefix)     descend(item, deepness + 1)   end end  def descend(item, deepness)   item.search('> ol').each do |ol|     ol_rule(ol, deepness: deepness)   end   item.search('> ul').each do |ul|     ul_rule(ul, deepness: deepness)   end end  doc = Nokogiri::HTML.fragment(text)  doc.search('ol:root').each do |group|   binding.pry   ol_rule(group, deepness: 1) end  doc.search('ul:root').each do |group|   ul_rule(group, deepness: 1) end 

You can then remove the tags or use doc.inner_text depending on your environment.

Two caveats though:

  1. Your entry selector must be carefully selected. I used your snippet verbatim without root element, thus i had to use ul:root/ol:root. Maybe "body > ol" works for your situation too. Maybe selecting each ol/ul but than walking each and only find those, that have no list parent.
  2. Using your example verbatim, Nokogiri does not handle the last 2 list items of the first group ol very well ("hey", "Three") When parsing with nokogiri, thus elements already "left" their ol tree and got placed in the root tree.

Current Output:

  1. One   2. Two       a. Inner One       b. inner Two         ◦ hey         ◦ hey       3. hey       4. hey   hey   Three    1. Introduction     a. Introduction   2. Description   3. Observation   4. Results   5. Summary    • Introduction   • Description       ◦ Observation           * Results               - Summary   • Overview 

Answers 2

Whenever you are in a ol, li or ul element, you must recursively check for ol, li and ul. If there are none of them, return (what have been discovered as a substructure), if there are, call the same function on the new node and add its return value to the current structure.

You perform a different action on each node no matter where it is depending on its type and then the function automatically repackage everything.

Read More

Sunday, July 9, 2017

Slower while generating the XML from the bunch of model object

Leave a Comment
class GenericFormatter < Formatter  attr_accessor :tag_name,:objects   def generate_xml    builder = Nokogiri::XML::Builder.new do |xml|    xml.send(tag_name.pluralize) {    objects.each do |obj|         xml.send(tag_name.singularize){              self.generate_obj_row obj,xml         }                     end     }    end    builder.to_xml  end   def initialize tag_name,objects   self.tag_name = tag_name   self.objects = objects end   def generate_obj_row obj,xml    obj.attributes.except("updated_at").map do |key,value|      xml.send(key, value)    end    xml.updated_at obj.updated_at.try(:strftime,"%m/%d/%Y %H:%M:%S") if obj.attributes.key?('updated_at') end  end  

In the above code, I have implemented the formatter where I have used the nokogiri XML Builder to generate the XML by manipulating the objects passing out inside the code.It's generated the faster XML when the data is not too large if data is larger like more than 10,000 records then It's slow down the XML to generate and takes at least 50-60 seconds.

Problem: Is there any way to generate the XML faster, I have tried XML Builders on view as well but did n't work.How can I generate the XML Faster? Should the solution be an application on rails 3 and suggestions to optimized above code?

2 Answers

Answers 1

Your main problem is processing everything in one go instead of splitting your data into batches. It all requires a lot of memory, first to build all those ActiveRecord models and then to build memory representation of the whole xml document. Meta-programming is also quite expensive (I mean those send methods).

Take a look at this code:

class XmlGenerator   attr_accessor :tag_name, :ar_relation    def initialize(tag_name, ar_relation)     @ar_relation = ar_relation     @tag_name = tag_name   end    def generate_xml     singular_tag_name = tag_name.singularize     plural_tag_name = tag_name.pluralize      xml = ""     xml << "<#{plural_tag_name}>"      ar_relation.find_in_batches(batch_size: 1000) do |batch|       batch.each do |obj|         xml << "<#{singular_tag_name}>"          obj.attributes.except("updated_at").each do |key, value|           xml << "<#{key}>#{value}</#{key}>"         end          if obj.attributes.key?("updated_at")           xml << "<updated_at>#{obj.updated_at.strftime('%m/%d/%Y %H:%M:%S')}</updated_at>"         end          xml << "</#{singular_tag_name}>"       end     end      xml << "</#{tag_name.pluralize}>"     xml   end end  # example usage XmlGenerator.new("user", User.where("age < 21")).generate_xml 

Major improvements are:

  • fetching data from database in batches, you need to pass ActiveRecord collection instead of array of ActiveRecord models
  • generating xml by constructing strings, this has a risk of producing invalid xml, but it is much faster than using builder

I tested it on over 60k records. It took around 40 seconds to generate such xml document.

There is much more that can be done to improve this even further, but it all depends on your application.

Here are some ideas:

  • do not use ActiveRecord to fetch data, instead use lighter library or plain database driver
  • fetch only data that you need
  • tweak batch size
  • write generated xml directly to a file (if that is your use case) to save memory

Answers 2

The Nokogiri gem has a nice interface for creating XML from scratch, Nokogiri is a wrapper around libxml2.

Gemfile gem 'nokogiri' To generate xml simple use the Nokogiri XML Builder like this

xml = Nokogiri::XML::Builder.new { |xml|      xml.body do         xml.test1 "some string"         xml.test2 890         xml.test3 do             xml.test3_1 "some string"         end         xml.test4 "with attributes", :attribute => "some attribute"         xml.closing     end }.to_xml 

output

<?xml version="1.0"?> <body>   <test1>some string</test1>   <test2>890</test2>   <test3>     <test3_1>some string</test3_1>   </test3>   <test4 attribute="some attribute">with attributes</test4>   <closing/> </body> 

Demo: http://www.jakobbeyer.de/xml-with-nokogiri

Read More

Friday, April 28, 2017

How do I use a Rails cache to store Nokogiri objects?

Leave a Comment

I'm using Rails 5 to use a Rails cache to store Nokogiri objects.

I created this in config/initializers/cache.rb:

$cache = ActiveSupport::Cache::MemoryStore.new 

and I wanted to store documents like:

$cache.fetch(url) {   result = get_content(url, headers, follow_redirects) } 

but I'm getting this error:

Error during processing: (TypeError) no _dump_data is defined for class Nokogiri::HTML::Document /Users/davea/.rvm/gems/ruby-2.4.0/gems/activesupport-5.0.2/lib/active_support/cache.rb:671:in `dump' /Users/davea/.rvm/gems/ruby-2.4.0/gems/activesupport-5.0.2/lib/active_support/cache.rb:671:in `dup_value!' /Users/davea/.rvm/gems/ruby-2.4.0/gems/activesupport-5.0.2/lib/active_support/cache/memory_store.rb:128:in `write_entry' /Users/davea/.rvm/gems/ruby-2.4.0/gems/activesupport-5.0.2/lib/active_support/cache.rb:398:in `block in write' /Users/davea/.rvm/gems/ruby-2.4.0/gems/activesupport-5.0.2/lib/active_support/cache.rb:562:in `block in instrument' /Users/davea/.rvm/gems/ruby-2.4.0/gems/activesupport-5.0.2/lib/active_support/notifications.rb:166:in `instrument' /Users/davea/.rvm/gems/ruby-2.4.0/gems/activesupport-5.0.2/lib/active_support/cache.rb:562:in `instrument' /Users/davea/.rvm/gems/ruby-2.4.0/gems/activesupport-5.0.2/lib/active_support/cache.rb:396:in `write' /Users/davea/.rvm/gems/ruby-2.4.0/gems/activesupport-5.0.2/lib/active_support/cache.rb:596:in `save_block_result_to_cache' /Users/davea/.rvm/gems/ruby-2.4.0/gems/activesupport-5.0.2/lib/active_support/cache.rb:300:in `fetch' /Users/davea/Documents/workspace/myproject/app/helpers/webpage_helper.rb:116:in `get_cached_content' /Users/davea/Documents/workspace/myproject/app/helpers/webpage_helper.rb:73:in `get_url' /Users/davea/Documents/workspace/myproject/app/services/abstract_my_object_finder_service.rb:29:in `process_data' /Users/davea/Documents/workspace/myproject/app/services/run_crawlers_service.rb:26:in `block (2 levels) in run_all_crawlers' /Users/davea/.rvm/gems/ruby-2.4.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `run_task' /Users/davea/.rvm/gems/ruby-2.4.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in `block (3 levels) in create_worker' /Users/davea/.rvm/gems/ruby-2.4.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop' /Users/davea/.rvm/gems/ruby-2.4.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker' /Users/davea/.rvm/gems/ruby-2.4.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch' /Users/davea/.rvm/gems/ruby-2.4.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker' 

What do I need to do in order to be able to store these objects in a cache?

2 Answers

Answers 1

Store the xml as string, not the object and parse them once you get them out of the cache.

Edit: response to comment

Cache this instead

nokogiri_object.to_xml 

Edit2: response to comment. Something along this lines. You will need to post more code if you want more specific help.

nokogiri_object = Nokogiri::XML(cache.fetch('xml_doc')) 

Edit3: Response to 'Thanks but what is the code for "Store serialized object in cache"? I thought the body of the "$cache.fetch(url) {" would take care of storing and then retrieiving things?'

cache.write('url', xml_or_serialized_nokogiri_string) 

Answers 2

User Nokogiri's Serialize functionality:

$cache = ActiveSupport::Cache::MemoryStore.new  noko_object = Nokogiri::HTML::Document.new  serial = noko_object.serialize $cache.write(url, serial) // Serialized Nokogiri document is now in store at the URL key. result = $cache.read(url)  noko_object = Nokogiri::HTML::Document.new(result) // noko_object is now the original document again :) 

Check out the documentation here for more information.

Read More

Monday, February 27, 2017

How to avoid “Invalid byte sequence” when looking for link with text using Nokogiri

Leave a Comment

I'm using Rails 5 with Ruby 4.2 and scanning a document that I parsed with Nokogiri, looking in a case insensitive way for a link with text:

a_elt = doc ? doc.xpath('//a').detect { |node| /link[[:space:]]+text/i === node.text } : nil  

After getting the HTML of my web page in content, I parse it into a Nokogiri doc using:

doc = Nokogiri::HTML(content)  

The problem is, I'm getting

ArgumentError invalid byte sequence in UTF-8 

on certain web pages when using the above regular expression.

2.4.0 :002 > doc.encoding  => "UTF-8"  2.4.0 :003 > doc.xpath('//a').detect { |node| /individual[[:space:]]+results/i === node.text } ArgumentError: invalid byte sequence in UTF-8     from (irb):3:in `==='     from (irb):3:in `block in irb_binding'     from /Users/davea/.rvm/gems/ruby-2.4.0@global/gems/nokogiri-1.7.0/lib/nokogiri/xml/node_set.rb:187:in `block in each'     from /Users/davea/.rvm/gems/ruby-2.4.0@global/gems/nokogiri-1.7.0/lib/nokogiri/xml/node_set.rb:186:in `upto'     from /Users/davea/.rvm/gems/ruby-2.4.0@global/gems/nokogiri-1.7.0/lib/nokogiri/xml/node_set.rb:186:in `each'     from (irb):3:in `detect'     from (irb):3     from /Users/davea/.rvm/gems/ruby-2.4.0@global/gems/railties-5.0.1/lib/rails/commands/console.rb:65:in `start'     from /Users/davea/.rvm/gems/ruby-2.4.0@global/gems/railties-5.0.1/lib/rails/commands/console_helper.rb:9:in `start'     from /Users/davea/.rvm/gems/ruby-2.4.0@global/gems/railties-5.0.1/lib/rails/commands/commands_tasks.rb:78:in `console'     from /Users/davea/.rvm/gems/ruby-2.4.0@global/gems/railties-5.0.1/lib/rails/commands/commands_tasks.rb:49:in `run_command!'     from /Users/davea/.rvm/gems/ruby-2.4.0@global/gems/railties-5.0.1/lib/rails/commands.rb:18:in `<top (required)>'     from bin/rails:4:in `require'     from bin/rails:4:in `<main>'  

Is there a way I can rewrite the above to automatically account for the encoding or weird characters and not flip out?

1 Answers

Answers 1

Your question may have already been answered before. Have you tried the methods from "Is there any way to clean a file of "invalid byte sequence in UTF-8" errors in Ruby?"?

Specifically before the detect block, try to remove the invalid bytes and control characters except new line:

doc.scrub!("") doc.gsub!(/[[:cntrl:]&&[^\n\r]]/,"") 

Remember, scrub! is a Ruby 2.1+ method.

Read More

Tuesday, August 30, 2016

Nokogiri XSLT transform using multiple source XML files

Leave a Comment

I want to translate XML using Nokogiri. I built an XSL and it all works fine. I ALSO tested it in Intellij. My data comes from two XML files.

My problem occurs when I try to get Nokogiri to do the transform. I can't seem to find a way to get it to parse multiple source files.

This is the code I am using from the documentation:

require 'Nokogiri'  doc1 = Nokogiri::XML(File.read('F:/transcoder/xslt_repo/core_xml.xml',)) xslt = Nokogiri::XSLT(File.read('F:/transcoder/xslt_repo/google.xsl'))  puts xslt.transform(doc1) 

I tried:

require 'Nokogiri'  doc1 = Nokogiri::XML(File.read('F:/transcoder/xslt_repo/core_xml.xml',)) doc2 = Nokogiri::XML(File.read('F:/transcoder/xslt_repo/file_data.xml',)) xslt = Nokogiri::XSLT(File.read('F:/transcoder/xslt_repo/test.xsl'))  puts xslt.transform(doc1,doc2) 

However it seems transform only takes one argument, so at the moment I am only able to parse half the data I need:

<?xml version="1.0"?> <package package_id="LB000001">   <asset_metadata>     <series_title>test asset 1</series_title>     <season_title>Number 1</season_title>     <episode_title>ET 1</episode_title>     <episode_number>1</episode_number>     <license_start_date>21-07-2016</license_start_date>     <license_end_date>31-07-2016</license_end_date>     <rating>15</rating>     <synopsis>This is a test asset</synopsis>   </asset_metadata>   <video_file>     <file_name/>     <file_size/>     <check_sum/>   </video_file>   <image_1>     <file_name/>     <file_size/>     <check_sum/>   </image_1> </package> 

How can I get this to work?

Edit:

This is the core_metadata.xml which is created via a PHP code block and the data comes from a database.

<?xml version="1.0" encoding="utf-8"?> <manifest task_id="00000000373">   <asset_metadata>     <material_id>LB111111</material_id>     <series_title>This is a test</series_title>     <season_title>This is a test</season_title>     <season_number>1</season_number>     <episode_title>that test</episode_title>     <episode_number>2</episode_number>     <start_date>23-08-2016</start_date>     <end_date>31-08-2016</end_date>     <ratings>15</ratings>     <synopsis>this is a test</synopsis>   </asset_metadata>   <file_info>     <source_filename>LB111111</source_filename>     <number_of_segments>2</number_of_segments>     <segment_1 seg_1_start="00:00:10.000" seg_1_dur="00:01:00.000"/>     <segment_2 seg_2_start="00:02:00.000" seg_2_dur="00:05:00.000"/> <conform_profile definition="hd" aspect_ratio="16f16">ffmpeg -i S_PATH/F_NAME.mp4 SEG_CONFORM 2&gt; F:/Transcoder/logs/transcode_logs/LOG_FILE.txt</conform_profile> <transcode_profile profile_name="xbox" package_type="tar">ffmpeg -f concat -i T_PATH/CONFORM_LIST TRC_PATH/F_NAME.mp4 2&gt; F:/Transcoder/logs/transcode_logs/LOG_FILE.txt</transcode_profile>     <target_path>F:/profiles/xbox</target_path>   </file_info> </manifest> 

The second XML (file_date.xml) is dynamically create during the trancode process by nokogiri:

<?xml version="1.0"?> <file_data>   <video_file>     <file_name>LB111111_xbox_230816114438.mp4</file_name>     <file_size>141959922</file_size>     <md5_checksum>bac7670e55c0694059d3742285079cbf</md5_checksum>   </video_file>   <image_1>     <file_name>test</file_name>     <file_size>test</file_size>     <md5_checksum>test</md5_checksum>   </image_1> </file_data> 

I managed to work around this issue by making a call to by hard coding the file_date.xml into the XSLT file:

<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> <xsl:output method="xml" indent="yes"/> <xsl:template match="/">     <package>         <xsl:attribute name="package_id">             <xsl:value-of select="manifest/asset_metadata/material_id"/>         </xsl:attribute>         <asset_metadata>             <series_title>                 <xsl:value-of select="manifest/asset_metadata/series_title"/>             </series_title>             <season_title>                 <xsl:value-of select="manifest/asset_metadata/season_title"/>             </season_title>             <episode_title>                 <xsl:value-of select="manifest/asset_metadata/episode_title"/>             </episode_title>             <episode_number>                 <xsl:value-of select="manifest/asset_metadata/episode_number"/>             </episode_number>             <license_start_date>                 <xsl:value-of select="manifest/asset_metadata/start_date"/>             </license_start_date>             <license_end_date>                 <xsl:value-of select="manifest/asset_metadata/end_date"/>             </license_end_date>             <rating>                 <xsl:value-of select="manifest/asset_metadata/ratings"/>             </rating>             <synopsis>                 <xsl:value-of select="manifest/asset_metadata/synopsis"/>             </synopsis>         </asset_metadata>         <video_file>             <file_name>                 <xsl:value-of select="document('file_data.xml')/file_data/video_file/file_name"/>             </file_name>             <file_size>                 <xsl:value-of select="document('file_data.xml')/file_data/video_file/file_size"/>             </file_size>             <check_sum>                 <xsl:value-of select="document('file_data.xml')/file_data/video_file/md5_checksum"/>             </check_sum>         </video_file>         <image_1>             <file_name>                 <xsl:value-of select="document('file_data.xml')/file_data/image_1/file_name"/>             </file_name>             <file_size>                 <xsl:value-of select="document('file_data.xml')/file_data/image_1/file_size"/>             </file_size>             <check_sum>                 <xsl:value-of select="document('file_data.xml')/file_data/image_1/md5_checksum"/>             </check_sum>         </image_1>     </package> </xsl:template> 

I then use Saxon to do the transform:

xslt = "java -jar C:/SaxonHE9-7-0-7J/saxon9he.jar #{temp}core_metadata.xml #{temp}#{profile}.xsl > #{temp}#{file_name}.xml"  system("#{xslt}") 

I would love to find a way to do this without having to hardcode the file_date.xml into the XSLT.

1 Answers

Answers 1

Merge XML Documents and Transform

You'll have to do a bit of work to combine the XML content prior to your XLS-Transformation. @the-Tin-Man has a nice answer to a similar question in the archives, which can be adapted for your use case.

Let's say we have the following sample content:

<!--a.xml--> <?xml version="1.0"?> <xml>   <packages>     <package>Data here for A</package>     <package>Another Package</package>     </packages> </xml> <!--a.xml-->  <!--b.xml--> <?xml version="1.0"?> <xml>   <packages>     <package>B something something</package>     </packages> </xml> <!--end b.xml--> 

And we want to apply the following XLST template:

<!--transform.xslt--> <?xml version="1.0"?> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> <xsl:template match="//packages">   <html>   <body>     <h2>Packages</h2>     <ol>       <xsl:for-each select="./package">         <li><xsl:value-of select="text()"/></li>       </xsl:for-each>     </ol>   </body>   </html> </xsl:template> </xsl:stylesheet> <!--end transform.xslt--> 

If we have parallel document structure, as in this case, we can merge the two XML documents' content together and pass that along for transformation.

require 'Nokogiri'  doc1 = Nokogiri::XML(File.read('./a.xml')) doc2 = Nokogiri::XML(File.read('./b.xml'))  moved_packages = doc2.search('package') doc1.at('/descendant::packages[1]').add_child(moved_packages)  xslt = Nokogiri::XSLT(File.read('./transform.xslt'))  puts xslt.transform(doc1) 

This would generate the following output:

<html><body> <h2>Packages</h2> <ol> <li>Data here for A</li> <li>Another Package</li> <li>B something something</li> </ol> </body></html> 

If your XML documents have varying structure, you may benefit from an intermediary XML nodeset that you add your content to, rather than the shortcut of merging document 2 content into document 1.

Read More

Monday, April 25, 2016

How to add a namespace to existing xml file

Leave a Comment

I want to open this file and get all elements that start with us-gaap.

ftp://ftp.sec.gov/edgar/data/916789/0001558370-15-001143.txt 

To get elements I tried like this:

str = '<html><body><us-gaap:foo>foo</us-gaap:foo></body></html>' doc = Nokogiri::XML(File.read(str)) doc.xpath('//us-gaap:*') Nokogiri::XML::XPath::SyntaxError: Undefined namespace prefix: //us-gaap:* from /Users/ironsand/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/searchable.rb:165:in `evaluate' 

doc.namespaces returns {}, so I think I have to add namespace us-gaap.

There are some questions about "adding namespace with Nokogiri", but it looks like about how to create a new XML document, not how to add a namespace to existing documents.

How can I add a namespace to existing document?

I know I can remove the namespace by Nokogiri::XML::Document#remove_namespaces!, but I don't want to use it because it removes also necesarry information.

3 Answers

Answers 1

You have asked an XY Problem. You think that the problem is that you need to add a missing namespace; the real problem is that the file you're trying to parse is not valid XML.

require 'nokogiri' doc = Nokogiri.XML( IO.read('0001558370-15-001143.txt') ) doc.errors.length #=> 5716 

For example, the <ACCEPTANCE-DATETIME> 'element' opened on line 3 is never closed, and on line 16 there is a raw ampersand in the text:
STANDARD INDUSTRIAL CLASSIFICATION: ELECTRIC HOUSEWARES & FANS [3634]
which ought to be escaped as an entity.

However, the document has valid XML fragments within it! In particular, there is one XML document that defines xmlns:us-gaap namespace, from lines 27243-49312. Let's extract just that, using only the knowledge that the root element defines the namespace we want, and the assumptions that no element with the same name is nested within the document, and that the root element does not have an unescaped > character in any attribute. (These assumptions are valid for this file, but may not be valid for every XML file.)

txt = IO.read('0001558370-15-001143.txt') gaap_finder = %r{(<(\w+) [^>]+xmlns:us-gaap=.+?</\2>)}m txt.scan(gaap_finder) do |xml,_|   doc = Nokogiri.XML( xml )   gaaps = doc.xpath('//us-gaap:*')   p gaaps.length   #=> 569 end 

The code above handles the case where there may be more than one XML document in the txt file, though in this case there is only one.

Decoded, the gaap_finder regex says this:

  • %r{...}m — this is a regular expression (that allows slashes in it, unescaped) with "multiline mode", where a period will match newline characters
  • (...) — capture everything we find
  • < — start with a literal "less-than" symbol
  • (\w+) — find one or more word characters (the tag name), and save them
  • — the word characters must be followed by a space (important to avoid capturing the <xsd:xbrl ...> element in this file)
  • [^>]+ — followed by one or more characters that is NOT a "greater-than" symbol (to ensure that we stay in the same element that we started in)
  • xmlns:us-gaap\s*= — followed by this literal namespace declaration (which may have whitespace separating it from the equals sign)
  • .+? — followed by anything (as little as possible)...
  • </\2> — ...up until you see a closing tag with the same name as what we captured for the name of the starting tag

Because of the way scan works when the regex has capturing groups, each result is a two-element array, where the first element is the entire captured XML and the second element is the name of the tag that we captured (which we "discard" by assigning it to the _ variable).


If you want to be less magic about your capturing, the text file format appears to always wrap each XML document in <XBRL>...</XBRL>. So, you could do this to process every XML file (there are seven, five of which do not happen to have any us-gaap namespaces):

txt   = IO.read('0001558370-15-001143.txt') xbrls = %r{(?<=<XBRL>).+?(?=</XBRL>)}m      # find text inside <XBRL>…</XBRL> txt.scan(xbrls) do |xml|   doc = Nokogiri.XML( xml )   if doc.namespaces["xmlns:us-gaap"]     gaaps = doc.xpath('//us-gaap:*')     p gaaps.length   end end #=> 569 #=> 0        (for the XML Schema document that defines the namespace) 

Answers 2

I couldn't figure out how to update an existing doc with a new namespace, but since Nokogiri will recognize namespaces on the root element, and those namespaces are, syntactically, just attributes, you can update the document with a new namespace declaration, serialize the doc to a string, and re-parse it:

str = '<html><body><us-gaap:foo>foo</us-gaap:foo></body></html>' doc_without_ns = Nokogiri::XML(str) doc_without_ns.root['xmlns:us-gaap'] = 'http://your/actual/ns/here' doc = Nokogiri::XML(doc_without_ns.to_xml) doc.xpath("//us-gaap:*") # Returns [#<Nokogiri::XML::Element:0x3ff375583f9c name="foo" namespace=#<Nokogiri::XML::Namespace:0x3ff375583f24 prefix="us-gaap" href="http://your/actual/ns/here"> children=[#<Nokogiri::XML::Text:0x3ff375583768 "foo">]>] 

Answers 3

I think you can refer to w3school also below is the site :- http://www.w3schools.com/xml/xml_namespaces.asp

Read More