Sunday, December 31, 2017

FAILURE: Build failed with an exception. on existing react native app

Leave a Comment

I am facing a issue with React Native existing app code during installation in my linux OS.

Everything is setup (React Native and Android Studio )

I am facing when i am running a command

here is the error message,

:app:processDebugResources /home/akaruilabs/ReactNative/teletask/android/app/build/intermediates/res/merged/debug/values-v24/values-v24.xml:3: AAPT: Error retrieving parent for item: No resource found that matches the given name 'android:TextAppearance.Material.Widget.Button.Borderless.Colored'.  /home/akaruilabs/ReactNative/teletask/android/app/build/intermediates/res/merged/debug/values-v24/values-v24.xml:4: AAPT: Error retrieving parent for item: No resource found that matches the given name 'android:TextAppearance.Material.Widget.Button.Colored'.  /home/akaruilabs/ReactNative/teletask/android/app/build/intermediates/res/merged/debug/values-v24/values-v24.xml:3: error: Error retrieving parent for item: No resource found that matches the given name 'android:TextAppearance.Material.Widget.Button.Borderless.Colored'.  /home/akaruilabs/ReactNative/teletask/android/app/build/intermediates/res/merged/debug/values-v24/values-v24.xml:4: error: Error retrieving parent for item: No resource found that matches the given name 'android:TextAppearance.Material.Widget.Button.Colored'.   :app:processDebugResources FAILED  FAILURE: Build failed with an exception.  * What went wrong: Execution failed for task ':app:processDebugResources'. > com.android.ide.common.process.ProcessException: Failed to execute aapt  * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.  BUILD FAILED  Total time: 16.116 secs Could not install the app on the device, read the error above for details. Make sure you have an Android emulator running or a device connected and have set up your Android development environment: https://facebook.github.io/react-native/docs/android-setup.html [bg] Exited with code 0 

Here is my build.gradle file

apply plugin: "com.android.application"  import com.android.build.OutputFile  /**  * The react.gradle file registers a task for each build variant (e.g. bundleDebugJsAndAssets  * and bundleReleaseJsAndAssets).  * These basically call `react-native bundle` with the correct arguments during the Android build  * cycle. By default, bundleDebugJsAndAssets is skipped, as in debug/dev mode we prefer to load the  * bundle directly from the development server. Below you can see all the possible configurations  * and their defaults. If you decide to add a configuration block, make sure to add it before the  * `apply from: "../../node_modules/react-native/react.gradle"` line.  *  * project.ext.react = [  *   // the name of the generated asset file containing your JS bundle  *   bundleAssetName: "index.android.bundle",  *  *   // the entry file for bundle generation  *   entryFile: "index.android.js",  *  *   // whether to bundle JS and assets in debug mode  *   bundleInDebug: false,  *  *   // whether to bundle JS and assets in release mode  *   bundleInRelease: true,  *  *   // whether to bundle JS and assets in another build variant (if configured).  *   // See http://tools.android.com/tech-docs/new-build-system/user-guide#TOC-Build-Variants  *   // The configuration property can be in the following formats  *   //         'bundleIn${productFlavor}${buildType}'  *   //         'bundleIn${buildType}'  *   // bundleInFreeDebug: true,  *   // bundleInPaidRelease: true,  *   // bundleInBeta: true,  *  *   // the root of your project, i.e. where "package.json" lives  *   root: "../../",  *  *   // where to put the JS bundle asset in debug mode  *   jsBundleDirDebug: "$buildDir/intermediates/assets/debug",  *  *   // where to put the JS bundle asset in release mode  *   jsBundleDirRelease: "$buildDir/intermediates/assets/release",  *  *   // where to put drawable resources / React Native assets, e.g. the ones you use via  *   // require('./image.png')), in debug mode  *   resourcesDirDebug: "$buildDir/intermediates/res/merged/debug",  *  *   // where to put drawable resources / React Native assets, e.g. the ones you use via  *   // require('./image.png')), in release mode  *   resourcesDirRelease: "$buildDir/intermediates/res/merged/release",  *  *   // by default the gradle tasks are skipped if none of the JS files or assets change; this means  *   // that we don't look at files in android/ or ios/ to determine whether the tasks are up to  *   // date; if you have any other folders that you want to ignore for performance reasons (gradle  *   // indexes the entire tree), add them here. Alternatively, if you have JS files in android/  *   // for example, you might want to remove it from here.  *   inputExcludes: ["android/**", "ios/**"],  *  *   // override which node gets called and with what additional arguments  *   nodeExecutableAndArgs: ["node"]  *  *   // supply additional arguments to the packager  *   extraPackagerArgs: []  * ]  */  apply from: "../../node_modules/react-native/react.gradle"  /**  * Set this to true to create two separate APKs instead of one:  *   - An APK that only works on ARM devices  *   - An APK that only works on x86 devices  * The advantage is the size of the APK is reduced by about 4MB.  * Upload all the APKs to the Play Store and people will download  * the correct one based on the CPU architecture of their device.  */ def enableSeparateBuildPerCPUArchitecture = false  /**  * Run Proguard to shrink the Java bytecode in release builds.  */ def enableProguardInReleaseBuilds = false  android {     compileSdkVersion 23     buildToolsVersion "23.0.1"      defaultConfig {         applicationId "com.mynativeapp"         minSdkVersion 16         targetSdkVersion 22         versionCode 1         versionName "1.0"         ndk {             abiFilters "armeabi-v7a", "x86"         }     }     splits {         abi {             reset()             enable enableSeparateBuildPerCPUArchitecture             universalApk false  // If true, also generate a universal APK             include "armeabi-v7a", "x86"         }     }     buildTypes {         release {             minifyEnabled enableProguardInReleaseBuilds             proguardFiles getDefaultProguardFile("proguard-android.txt"), "proguard-rules.pro"         }     }     // applicationVariants are e.g. debug, release     applicationVariants.all { variant ->         variant.outputs.each { output ->             // For each separate APK per architecture, set a unique version code as described here:             // http://tools.android.com/tech-docs/new-build-system/user-guide/apk-splits             def versionCodes = ["armeabi-v7a":1, "x86":2]             def abi = output.getFilter(OutputFile.ABI)             if (abi != null) {  // null for the universal-debug, universal-release variants                 output.versionCodeOverride =                         versionCodes.get(abi) * 1048576 + defaultConfig.versionCode             }         }     } }  dependencies {     compile project(':react-native-splash-screen')     compile project(':react-native-navigation')     compile project(':react-native-vector-icons')     compile fileTree(dir: "libs", include: ["*.jar"])     compile "com.android.support:appcompat-v7:23.0.1"     compile "com.facebook.react:react-native:+"  // From node_modules }  // Run this once to be able to run the application with BUCK // puts all compile dependencies into folder libs for BUCK to use task copyDownloadableDepsToLibs(type: Copy) {     from configurations.compile     into 'libs' } 

And settings.gradle

rootProject.name = 'mynativeapp'  include ':app' include ':react-native-splash-screen' project(':react-native-splash-screen').projectDir = new File(rootProject.projectDir, '../node_modules/react-native-splash-screen/android') include ':react-native-navigation' project(':react-native-navigation').projectDir = new File(rootProject.projectDir, '../node_modules/react-native-navigation/android/app') include ':react-native-vector-icons' project(':react-native-vector-icons').projectDir = new File(rootProject.projectDir, '../node_modules/react-native-vector-icons/android') 

Terminal Error Image

enter image description here

I am not able to get what exactly error occurring here. Let me know for a solution.

1 Answers

Answers 1

Here are my 2 cents.

Potential fix:

  • Update your buildToolsVersion to 24.0.3 or later.
  • Update your compileSdkVersion 24 or to a newer version greater than 24.
  • Also update your appcompat library based on that. i.e compile "com.android.support:appcompat-v7:24.0.3"

P.S: You might need to download that build tools version from SDK manager. It is possible that it might not be available for download. In that case you can use the latest buildToolsVersion.

What is the potential issue:

Pay heed to these messages:

/home/akaruilabs/ReactNative/teletask/android/app/build/intermediates/res/merged/debug/values-v24/values-v24.xml:3: AAPT: Error retrieving parent for item: No resource found that matches the given name 'android:TextAppearance.Material.Widget.Button.Borderless.Colored'.

...

/home/akaruilabs/ReactNative/teletask/android/app/build/intermediates/res/merged/debug/values-v24/values-v24.xml:4: error: Error retrieving parent for item: No resource found that matches the given name 'android:TextAppearance.Material.Widget.Button.Colored'.

You are using buildToolsVersion "23.0.1" and your compileSdkVersion as 23. However you have a values-24 folder. It is a directory that contains resources that will be used when the device that is running your app is on API Level 24 or higher.

The compileSdkVersion is the version of the API the app is compiled against. This means you can use Android API features included in that version of the API (as well as all previous versions). If you try and use API 24 features but set compileSdkVersion to 23, you will get a compilation error. If you set compileSdkVersion to 24, you can still run the app on a API 23 device as long as your app's execution paths do not attempt to invoke any APIs specific to API 23.

The buildToolsVersion specifies the version of the SDK Build Tools to use when building your project. Just make sure it supports the compileSdkVersion that you are using. The newer versions of buildToolsVersion support the older compileSdkVersion versions.

Useful References:

Hope this helps.

Read More

Python fbprophet - export values from plot_components() for yearly

Leave a Comment

Any ideas how to export the yearly seasonal trend using fbprophet? The plot_components() function plots the trend, yearly, and weekly. I want to obtain the values for yearly only.

Much appreciated.

1 Answers

Answers 1

Example data:

import pandas as pd from fbprophet import Prophet import matplotlib.pyplot as plt  df = pd.DataFrame.from_dict({'ds': ['1949-01', '1949-02', '1949-03', '1949-04', '1949-05', '1949-06',         '1949-07', '1949-08', '1949-09', '1949-10', '1949-11', '1949-12',         '1950-01', '1950-02', '1950-03', '1950-04', '1950-05', '1950-06',         '1950-07', '1950-08', '1950-09', '1950-10', '1950-11', '1950-12',         '1951-01', '1951-02', '1951-03', '1951-04', '1951-05', '1951-06',         '1951-07', '1951-08', '1951-09', '1951-10', '1951-11', '1951-12',         '1952-01', '1952-02', '1952-03', '1952-04', '1952-05', '1952-06',         '1952-07', '1952-08', '1952-09', '1952-10', '1952-11', '1952-12',         '1953-01', '1953-02', '1953-03', '1953-04', '1953-05', '1953-06',         '1953-07', '1953-08', '1953-09', '1953-10', '1953-11',  '1953-12',         '1954-01', '1954-02', '1954-03', '1954-04', '1954-05', '1954-06',         '1954-07', '1954-08', '1954-09', '1954-10', '1954-11', '1954-12',         '1955-01', '1955-02', '1955-03', '1955-04', '1955-05', '1955-06',         '1955-07', '1955-08', '1955-09', '1955-10', '1955-11', '1955-12',         '1956-01', '1956-02', '1956-03', '1956-04', '1956-05', '1956-06',         '1956-07', '1956-08', '1956-09', '1956-10', '1956-11', '1956-12',         '1957-01', '1957-02', '1957-03', '1957-04', '1957-05', '1957-06',         '1957-07', '1957-08', '1957-09', '1957-10', '1957-11', '1957-12',         '1958-01', '1958-02', '1958-03', '1958-04', '1958-05', '1958-06',         '1958-07', '1958-08', '1958-09', '1958-10', '1958-11', '1958-12',         '1959-01', '1959-02', '1959-03', '1959-04', '1959-05', '1959-06',         '1959-07', '1959-08', '1959-09', '1959-10', '1959-11', '1959-12',         '1960-01', '1960-02', '1960-03', '1960-04', '1960-05', '1960-06',         '1960-07', '1960-08', '1960-09', '1960-10', '1960-11', '1960-12'],  'y': [112, 118, 132, 129, 121, 135, 148, 148, 136, 119, 104, 118, 115, 126,        141, 135, 125, 149, 170, 170, 158, 133, 114, 140, 145, 150, 178, 163,        172, 178, 199, 199, 184, 162, 146, 166, 171, 180, 193, 181, 183, 218,        230, 242, 209, 191, 172, 194, 196, 196, 236, 235, 229, 243, 264, 272,        237, 211, 180, 201, 204, 188, 235, 227, 234, 264, 302, 293, 259, 229,        203, 229, 242, 233, 267, 269, 270, 315, 364, 347, 312, 274, 237, 278,        284, 277, 317, 313, 318, 374, 413, 405, 355, 306, 271, 306, 315, 301,        356, 348, 355, 422, 465, 467, 404, 347, 305, 336, 340, 318, 362, 348,        363, 435, 491, 505, 404, 359, 310, 337, 360, 342, 406, 396, 420, 472,        548, 559, 463, 407, 362, 405, 417, 391, 419, 461, 472, 535, 622, 606,        508, 461, 390, 432]}) df['ds'] = pd.to_datetime(df['ds']) 

fbprophet plot_components():

model = Prophet() model.fit(df) future = model.make_future_dataframe(12, 'm') fc = model.predict(future) model.plot_components(fc) plt.show() 

prophet_plot_components

I understand your question to mean "How do we get the values used in the "yearly" plot above?

days = (pd.date_range(start='2017-01-01', periods=365) + pd.Timedelta(days=0)) df_y = model.seasonality_plot_df(days) seas = model.predict_seasonal_components(df_y)  fig,ax = plt.subplots(2, 1, figsize=(8,6)) ax[0].plot(fc['ds'].dt.to_pydatetime(), fc['trend']) ax[0].grid(alpha=0.5) ax[0].set_xlabel('ds') ax[1].set_ylabel('trend') ax[1].plot(df_y['ds'].dt.to_pydatetime(), seas['yearly'], ls='-', c='#0072B2') ax[1].set_xlabel('Day of year') ax[1].set_ylabel('yearly') ax[1].grid(alpha=0.5) plt.show() 

Plot results:

results plotted manually

How to obtain the values for yearly only?

The X values are an arbitrary yearly time period (basically). The y values use functions seasonality_plot_df() and predict_seasonal_components() to predict daily seasonality for a year time-span. You retrieve these values by looking in seas['yearly'].

Read More

Does not login into account at first time in Codeigniter 3 framework

Leave a Comment

I have a website and login system. In localhost everything is perfect. However, in the server (HOSTGATOR), when people want to log in their account, they type login password and hit "Login" button, it refreshes the page but does not log in. They should enter 2-3 times and then systems logs in. Where can be the problem?

public function __construct() {     parent::__construct();     $this->load->helper(array('form','url', 'security'));     $this->load->library(array('session', 'form_validation', 'email'));     $this->load->database();     $this->load->model('User_model'); }   public function login(){             $data['title'] = 'Sign In';          $validator = array('success' => false, 'messages' => array());          $validate_data = array(             array(                 'field' => 'usernameforlog',                 'label' => lang('controller-username-label'),                 'rules' => 'trim|required|alpha_dash'             ),             array(                 'field' => 'passwordforlog',                 'label' => lang('controller-password-label'),                 'rules' => 'trim|required|md5'             )         );          $this->form_validation->set_rules($validate_data);         $this->form_validation->set_message('required', lang("form_validation_required"));           $this->form_validation->set_message('alpha_dash', lang("form_validation_alpha_dash"));         $this->form_validation->set_error_delimiters('<p class="text-danger">', '</p>');                if ($this->form_validation->run() === FALSE)             {                  // fails                 $validator['success'] = false;                 foreach ($_POST as $key => $value) {                     $validator['messages'][$key] = form_error($key);                 }                  log_message('error', 'validation errors' );              }else {                  // Get username                 $username = mb_strtolower($this->input->post('usernameforlog'), 'UTF-8');                 // Get and encrypt the password                 $password = $this->input->post('passwordforlog');                 // Login user                  $user_id = $this->user_model->login($username, $password);                 if($user_id){                     // Create session                      $user_data = array(                      'id' => $user_id,                      'instructors_slug' => $username,                      'logged_in' => true                      );                      $this->session->set_userdata($user_data);                     $validator['success'] = true;                     $validator['messages'] = array();                 } else {                      $validator['success'] = false;                     $validator['messages'] = '<div class="alert alert-danger text-center">'.lang('controller-user-email-error3').'</div>';                 }                    }              echo json_encode($validator);         } 

Here is the ajax file:

$(document).ready(function() {     $("#loginform").unbind('submit').bind('submit', function() {         var form = $(this);          $.ajax({             url: form.attr('action'),             type: form.attr('method'),             data: form.serialize(),             dataType: 'json',             success:function(response) {             console.log(response);                               if(response.success) {                     //redirect main page                     location.reload();                 }                 else {                     $("#loginmsg").html(response.messages);                      $.each(response.messages, function(index, value) {                         var element = $("#"+index);                          $(element)                         .closest('.form-group')                         .removeClass('has-error')                         .removeClass('has-success')                         .addClass(value.length > 0 ? 'has-error' : 'has-success')                         .find('.text-danger').remove();                          $(element).after(value);                      });                 }             } // /success         });  // /ajax          return false;     }); }); 

1 Answers

Answers 1

because location.reload() will reload your login() page, in here you need to add method for checking if user has logged_in ?

public function login() {      $logged_in = $this->session->userdata('logged_in');      if($logged_in)     {         // example redirect to dashboard         redirect('dashboard', 'refresh');     }       // rest of code } 

instead just echo json_encode($validator); better set also your content header as application/json

return $this->output->set_content_type('application/json')->set_output(json_encode($this->response)); 
Read More

How to trigger MouseDown via PhantomJS?

Leave a Comment

I tried:

var clickEvent = document.createEvent('MouseEvents'); clickEvent.initEvent("mousedown", true, true); jQuery('.left-rail-facets .facet-list .facet.CS .suggestion[data-value="B"] a')[0].dispatchEvent(clickEvent); 

but it just fails to trigger the action on the site. no error returned.

2 Answers

Answers 1

This is how you programmatically trigger a click on a DOM element with a "mousedown" event. To answer your question, you need to have two things in your source code:

event listeners to detect the clicks

and

commas between your multiple selectors in your jQuery line.

Please run the snippet below, which shows the modified jQuery line and the listeners running correctly.

<!DOCTYPE html>  <html>    <head>    <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>    <script>      $(document).ready(function() {        // You must have event listeners to see if anything is actually happening. For example, here I will add only a listener to the first element:        jQuery(".left-rail-facets")[0].addEventListener('mousedown', function(e) {          jQuery(".left-rail-facets")[0].innerHTML = "I have received a click.";        }, false);        // Now here is the code from your question. Please see how I have commas instead of spaces in my jQuery selector:        var clickEvent = document.createEvent('MouseEvents');       clickEvent.initEvent('mousedown', true, true);        jQuery('.left-rail-facets,.facet-list,.facet.CS,.suggestion[data-value="B"],a')[0].dispatchEvent(clickEvent);      });    </script>  </head>    <body>      <h1>I am a Webpage</h1>      <p class="left-rail-facets">I have not received a click.</p>    <p class="facet-list">I have not received a click.</p>    <facet class="CS">I have not received a click.</facet>    <p class="suggestion" data-value="B">I have not received a click.</p>    <a href="url">I have not received a click.</a>    </body>    </html>

Answers 2

You will need to use the document.createEvent and event.initMouseEvent:

var element = document.querySelector('.left-rail-facets .facet-list .facet.CS .suggestion[data-value="B"] a'); //or var element = jQuery('.left-rail-facets .facet-list .facet.CS .suggestion[data-value="B"] a')[0]; var elementRect = element.getBoundingClientRect(); var mouseEvent = document.createEvent('MouseEvents'); mouseEvent.initMouseEvent('mousedown', true, true, window, 0, elementRect.left, elementRect.top); 
Read More

Retrieve data from a table of aspx page using Excel VBA

Leave a Comment

I am trying to retrieve table data from aspx page using excel vba.I know how to get table data from a URL but below is the main problem.

Problem

There is an aspx page (say www.abc.aspx). I am currently on this page.Let this page be page1.

Now I click a page2 link on the current page. What is worth noticing is that after clicking this link, the old URL (www.abc.aspx) doesn't change but the content changes.( Content is of page2 )

If you view page1 source code it has

<form method="post" action="page1 url" id="Form1"> 

Whatever is the action on page1 (page2 click) , it posts back the same page1 url.

So how can I get page2 table data in excel VBA since I don't know its URL?

Code

This is what I had used to fetch table data.

I used internet explorer object.Then navigated to the link and saved the document in htmldoc.

ie.navigate "url"  Do While ie.READYSTATE <> READYSTATE_COMPLETE Application.StatusBar = "Fetching data..." DoEvents Loop  Set htmldoc = ie.document  'Column headers Set eleColth = htmldoc.getElementsByTagName("th") j = 0 'start with the first value in the th collection         For Each eleCol In eleColth 'for each element in the td collection             ThisWorkbook.Sheets(1).Range("A1").Offset(i, j).Value = eleCol.innerText 'paste the inner text of the td element, and offset at the same time             j = j + 1 'move to next element in td collection         Next eleCol 'rinse and repeat   'Content Set eleColtr = htmldoc.getElementsByTagName("tr")  'This section populates Excel     i = 0 'start with first value in tr collection     For Each eleRow In eleColtr 'for each element in the tr collection         Set eleColtd = htmldoc.getElementsByTagName("tr")(i).getElementsByTagName("td") 'get all the td elements in that specific tr         j = 0 'start with the first value in the td collection         For Each eleCol In eleColtd 'for each element in the td collection             ThisWorkbook.Sheets(1).Range("D3").Offset(i, j).Value = eleCol.innerText 'paste the inner text of the td element, and offset at the same time             j = j + 1 'move to next element in td collection         Next eleCol 'rinse and repeat         i = i + 1 'move to next element in td collection     Next eleRow 'rinse and repeat  ie.Quit Set ie = Nothing 

EDIT:

Example

If we click on questions in Stack Overflow (https://stackoverflow.com/questions) and now click on page2 of questions (new link is https://stackoverflow.com/questions?page=2&sort=newest)

In my case, if we click on page2, the new link is not updated.It is the same old link.

EDIT: I have found a similar question here

How do I get url that is hidden by javascript on external website?

Thanks.

2 Answers

Answers 1

Ok, I sympathise, there is a school of thought (including Tim Berners-Lee) that says every separate page should have its own URI and that these don't change.

But webmasters can and do mess you around. They can redirect you HTTP request and can obfuscate the navigation as in your case. They can rewrite HTTP requests.

You have two options

Option 1 - Let Internet Explorer resolve the new content for you

So, if the content is visible on the screen then it must be in the Document Object Model (DOM). In IE, or indeed in Chrome, one can right-click and get the context menu and then choose Inspect to see where in the DOM that element resides.

I think your code demonstrates enough expertise to drill in. However, sometimes some websites like to disable the Inspect menu option to avoid programers poking around. (EDIT: As in your case now that I have read the comments)

Option 2 - Use an HTTP sniffing Tool like Fiddler to detect the HTTP redirect/rewrite

As I said above, HTTP requests can be rewritten and redirected by the web-server but the HTTP protocol does give notifications of redirects. There are tools to detect this. A popular tool is Fiddler, today I have discovered that there is a specific IE Fiddler add-on.

To be honest though the developer tools that ship with the browsers themselves, particularly Chrome (Ctrl+Shift+I, then Network tab), show network traffic to a level of detail increasingly on a par with any sniffing tool.

Sorry you got down-voted, this seems like a perfectly reasonable question.

Answers 2

A bird's eye view on the problem:

You have a requirement that you seem to not be able to let go: Use Excel VBA. I emphasize this point since often times answers provide solutions satisfying alternative premises from what is posted in the OP.

A possible solution:

So you have to Interface Excel VBA with another tool having the capability for revealing the contents of html redirects or obfuscated URLs.

Google Chrome Developer Tools reveals all contents, and you can interface very nicely Google Chrome with Excel VBA, using the Selenium VBA Wrapper. Download here.

It is quite versatile, e.g., you can see how to scrape web data.

As for getting obfuscated contents, there are a few items that may help

how to get innerHTML of whole page in selenium driver? (not VBA but useful)

Selenium + VBA to Control Chrome

(Note: the author of the wrapper is usually eager to answer in SO, and precise in his answers).

I guess YMMV, there are always people trying to obfuscate their data, with various techniques, and often for good reasons...

If you have a real example for your http://www.abc.aspx, it may help.

Read More

HTML5 Video - Get final src URL destination

Leave a Comment

I'm displaying video content from a 3rd party service on my website:

<video src="http://www.example.com/api/v1/media/video.mp4?status=true"></video> 

The service detects the user's geo-location and browser's language and returns video that fit those parameters.

http://www.example.com/api/v1/media/video.mp4?status=true REDIRECT TO http://www.example.com/api/v1/media/US/EN/English_CAT.mp4 

Other than the media URL, the service doesn't provide the actual file name that particular user received.

How can I fetch the final media path/name? Tried multiple ways but iframes blocked by X-Frame-Options: DENY and AJAX blocked as well.

1 Answers

Answers 1

The short answer is you probably can't do this all on the client-side. You will run into same-origin issues. In addition, it is not possible to directly get the HTTP headers of a <video> element.

If the service provider for the videos enables CORS support you may be able to get the final URL via a workaround. Even with CORS enabled you won't be able to get the HTTP headers or final URL of the video from the <video> element.

You can work around this by:

  1. Make a 2nd HTTP request (XMLHttpRequest, fetch, $.ajax, etc) to the same URL.
  2. Make a HEAD request instead of a GET request. GET requests will attempt to retrieve the entire file. HEAD requests do not download the body, only the headers, which would be enough for resolving the redirect.
  3. Look in the headers of the HEAD response for the final URL path.

Here is an example that uses fetch:

var request = new Request(yourVideoUrl); request.mode = 'cors'; request.method = 'HEAD';  fetch(request) .then(function(res){   console.log(res.url); // This should be the final URL }); 
Read More

Saturday, December 30, 2017

Selenium hangs when using actionchain().move then actionchain.click or mouse_up

Leave a Comment

I'm using selenium2library for automation testing for drag drop action. I'm running on windows 8 64bit, selenium 2.48.0, ride.py. Browser used to test: firefox and chrome latest stable version

what I did was to create a dummy html page with input text and a link, and try drag that link into the input text

Here is the html:

<div id="wrapper">  <input id="target" type="text" style="width:200px; height:50px" /> </div> <a id="source" href="http://google.com" >drag me </a> 

And here is my python code for automation:

class CustomSeleniumLibrary(Selenium2Library): ...     def test_drag(self):         self.open_browser("http://localhost:8080/a.html", "firefox")         source = self._element_find("//a[@id='source']", True, True)         target = self._element_find("//input[@id='target']", True, True)         drag = ActionChains(self._current_browser()).click_and_hold(source)         moveDum = ActionChains(self._current_browser()).move_by_offset(1,1)         move = ActionChains(self._current_browser()).move_to_element_with_offset(target,1,1)         #I have also tried ActionChains().drag_and_drop().perform() or make a dummy move move_by_offset followed by move_to_element_with_offset but no use         drag.perform()         moveDum.perform()        move.perform() 

What I found is when the move finish or mouse_down() finish, the next action is not performed, I can see the link get holded, but no move action performs until I manually move my mouse on the browser. ride.py UI flicks at that time and the request: 16:24:47.042 : DEBUG : POST http://127.0.0.1:58095/hub/session/fa7590b6-396f-4cb5-a08a-e35138a9216e/moveto {"sessionId": "fa7590b6-396f-4cb5-a08a-e35138a9216e", "element": "{6586b4ae-3c51-4e18-bb40-e006af369768}", "xoffset": 1, "yoffset": 1}

hangs forever until I move the mouse manually on the browser

Do anyone of you got the same problem, or did I do something wrong? And do you have any suggestion for using draganddrop feature using robotframework selenium2library?

Best regards, Dan

1 Answers

Answers 1

I cannot check it, but as I remember ActionChains works in this way:

actions = ActionChains(self._current_browser()) actions.click_and_hold(source) actions.move_by_offset(1,1) actions.move_to_element_with_offset(target,1,1) actions.perform() 

Let me know if this code works incorrectly

Read More

React Native: Is it possible to create floating chat heads like facebook messenger?

Leave a Comment

Is it possible to create floating chat heads like facebook messenger by using react native?

1 Answers

Answers 1

Without native coding, you can just make them show inside of your app, here is fine example, it basically extends react-native-interactible example.

If you want them to stay when the app is closed, I am afraid that there is not some react native library at the moment (as far as I know) that can provide you that, just native code something like in this article.

Read More

Twitter Boostrap - Align label horizontally to dropdown

Leave a Comment

I would like to have the label for the dropdown next to the dropdown menus.

Find below how my current example looks like:

<link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">  <ul class="nav nav-pills" role="tablist">    <li role="presentation">Order: </li>    <li role="presentation" class="dropdown">      <a href="#" class="dropdown-toggle" id="drop4" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Ticker<span class="caret"></span></a>      <ul class="dropdown-menu" id="menu1" aria-labelledby="drop4">        <li>          <a href="#">Action</a>        </li>        <li>          <a href="#">Another action</a>        </li>        <li>          <a href="#">Something else here</a>        </li>        <li role="separator" class="divider"></li>        <li>          <a href="#">Separated link</a>        </li>      </ul>    </li>  </ul>

Any suggestions how to align the label with the dropdown properly?

Thx in advance!

5 Answers

Answers 1

I'm not sure I understand correctly, but for alignment I would use a flexbox.

.nav-pills {    display: flex;    align-items: center;  }
<link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" rel="stylesheet" />  <ul class="nav nav-pills" role="tablist">    <li role="presentation">Order: </li>    <li role="presentation" class="dropdown">      <a href="#" class="dropdown-toggle" id="drop4" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">                              Ticker                                  <span class="caret"></span>                              </a>      <ul class="dropdown-menu" id="menu1" aria-labelledby="drop4">        <li>          <a href="#">Action</a>        </li>        <li>          <a href="#">Another action</a>        </li>        <li>          <a href="#">Something else here</a>        </li>        <li role="separator" class="divider"></li>        <li>          <a href="#">Separated link</a>        </li>      </ul>    </li>

Answers 2

Just add the below css in your custom css file

.nav-pills>li:not(.dropdown) {     padding: 10px 12px; } 

.nav-pills>li:not(.dropdown) {    padding: 10px 12px;  }
<link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">  <ul class="nav nav-pills" role="tablist">    <li role="presentation">Order: </li>    <li role="presentation" class="dropdown">      <a href="#" class="dropdown-toggle" id="drop4" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Ticker<span class="caret"></span></a>      <ul class="dropdown-menu" id="menu1" aria-labelledby="drop4">        <li>          <a href="#">Action</a>        </li>        <li>          <a href="#">Another action</a>        </li>        <li>          <a href="#">Something else here</a>        </li>        <li role="separator" class="divider"></li>        <li>          <a href="#">Separated link</a>        </li>      </ul>    </li>

Answers 3

You can simply add padding like this :

/* Optional theme */    @import url('//maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css');    ul.nav > li:first-child {   padding:10px 5px;  }
<ul class="nav nav-pills" role="tablist">    <li role="presentation" >Order: </li>    <li role="presentation" class="dropdown">      <a href="#" class="dropdown-toggle" id="drop4" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">                              Ticker                                  <span class="caret"></span>                              </a>      <ul class="dropdown-menu" id="menu1" aria-labelledby="drop4">        <li>          <a href="#">Action</a>        </li>        <li>          <a href="#">Another action</a>        </li>        <li>          <a href="#">Something else here</a>        </li>        <li role="separator" class="divider"></li>        <li>          <a href="#">Separated link</a>        </li>      </ul>    </li>

Answers 4

Use Flex to nav pills

.nav-pills{ display:flex; align-items:center; } 

JS FIDDLE: https://jsfiddle.net/pradeep0594/qfwngj5x/1/ for your reference

Answers 5

I like Gerard's display: flex solution better, but just in case anyone prefers to maintain the original display property, here is an example using only line-height to vertically center both .nav-pills.

<style>      .nav-pills > li {          line-height: 40px;      }      .nav.nav-pills > li > a {          /* removes top & bottom padding */          padding: 0 15px;      }  </style>    <link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">  <ul class="nav nav-pills" role="tablist">    <li role="presentation">Order: </li>    <li role="presentation" class="dropdown">      <a href="#" class="dropdown-toggle" id="drop4" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Ticker<span class="caret"></span></a>      <ul class="dropdown-menu" id="menu1" aria-labelledby="drop4">        <li>          <a href="#">Action</a>        </li>        <li>          <a href="#">Another action</a>        </li>        <li>          <a href="#">Something else here</a>        </li>        <li role="separator" class="divider"></li>        <li>          <a href="#">Separated link</a>        </li>      </ul>    </li>  </ul>

Read More

How to read nested properties from configuration file in java

Leave a Comment

I want to read Name property from configuration file

config.properties

Person{  Name= ABC   PhoneNumber=123   } Address   {   Pin=500  }  

I tried

public Properties readConfiguration() {     File configFile1 = new File("config.properties");      try {         FileReader reader = new FileReader(configFile1);         Properties props = new Properties();         props.load(reader);          reader.close();         return props;     } catch (FileNotFoundException ex) {         // file does not exist     } catch (IOException ex) {         // I/O error     }      return null; } 

But this function is not able to read nested properties.

Please help me find the solution.

3 Answers

Answers 1

your file config.properties structure looks like json format and to my understanding you want to parse a json file, correct me if i am wrong. To read or write from a json objects u can use Gson like this

my json file

{     Person:{      "Name" = "ABC",       "PhoneNumber" = 123       },     Address: {       "Pin" = 500      } } 

My POJO classes includes

    public class Person {         private String Name;         private  int PhoneNumber;          public Person() {         }          public String getName() {             return Name;         }          public void setName(String name) {             Name = name;         }          public int getPhoneNumber() {             return PhoneNumber;         }          public void setPhoneNumber(int phoneNumber) {             PhoneNumber = phoneNumber;         }     }     public class  Address{             private String Pin;              public Address() {             }              public String getPin() {                 return Pin;             }              public void setPin(String pin) {                 Pin = pin;             }         } public class Config{         private Person person;         private Address address;      public Config() {     }      public Person getPerson() {         return person;     }      public void setPerson(Person person) {         this.person = person;     }      public Address getAddress() {         return address;     }      public void setAddress(Address address) {         this.address = address;     } } 

read the file convert to string and then read json object using Gson.

 Scanner scanner = new Scanner(new File("config.properties"));         String fileStr="";         while (scanner.hasNextLine()){             fileStr+=scanner.nextLine();         }         Gson gson = new Gson();         Config config =gson.fromJson(fileStr,Config.class);         config.getPerson.getPhoneNumber(); 

Answers 2

It is not legal properties file. But it is legal HOCON conf.

An example of your conf file reading with hocon generated by tscfg.

import com.typesafe.config.Config; import com.typesafe.config.ConfigFactory; import org.junit.Assert;  import java.io.File;  public class SampleConf {     public final SampleConf.Address Address;     public final SampleConf.Person Person;      public static class Address {         public final int Pin;          public Address(final com.typesafe.config.Config c) {             this.Pin = c.hasPathOrNull("Pin") ? c.getInt("Pin") : 500;         }     }      public static class Person {         public final java.lang.String Name;         public final int PhoneNumber;          public Person(final com.typesafe.config.Config c) {             this.Name = c.hasPathOrNull("Name") ? c.getString("Name") : "ABC";             this.PhoneNumber = c.hasPathOrNull("PhoneNumber") ? c.getInt("PhoneNumber") : 123;         }     }      public SampleConf(final com.typesafe.config.Config c) {         this.Address = new SampleConf.Address(c.getConfig("Address"));         this.Person = new SampleConf.Person(c.getConfig("Person"));     }      public static void main(final String[] args) {          final Config config = ConfigFactory.parseFile(                 new File(("problem.so.hocon.conf"))) // file path                 .resolve();         final SampleConf conf = new SampleConf(config);         // do everything  you like         final SampleConf.Address address = conf.Address;         final SampleConf.Person person = conf.Person;         Assert.assertEquals(500, address.Pin);     }  } 

Answers 3

In order to directly answer your question, you are not doing a proper config.properties file.

normally, a property file would look like this as per your example:

//File  Name=ABC Phonenumber=123456 pin=1234 

In your code, you should be fine by just doing the following after ***props.load(reader)*** (Note I am not copying all your code.)

System.out.println(props.getProperty("Name")); // ABC System.out.println(props.getProperty("Phonenumber")); // 123456 System.out.println(props.getProperty("pin")); // 123 

Here I am simply outputting the results but note you can do whatever you wish to do.

Read More

PHP imap_fetchbody can't get HTML a tag link url

Leave a Comment

I'm building a email read script, it read emails via imap, when i try to read email body using imap_fetchbody it only show texts, but email body have one link it seems not show,

This is my code

    $hostname = '{mail.test.com/notls}INBOX';     $username = 'info@test.com';     $password = 'testopw';      /* try to connect */     $inbox = imap_open($hostname,$username,$password) or die('Cannot connect to Email Server: ' . imap_last_error());     /* grab emails */     $emails = imap_search($inbox,'SEEN FROM "noreply@test.com"'); if($emails) {      /* put the newest emails on top */     rsort($emails);     /* for every email... */     foreach($emails as $email_number) {         /* get information specific to this email */                        echo $message = imap_fetchbody($inbox,$email_number,1); } } 

The email contents is like below

 We do our utmost to arrive to you on time. There may, however,  be unexpected circumstances, causing our driver to be delayed.  You can follow the current status of your assignment here. 

When i read the email from mail client text here has a link, but when read that using php it only shows a text,

Email screen shot here enter image description here someone know what is the reason for this issue or is there need to change some parameters for get the link with url. Thank You

1 Answers

Answers 1

You have to check the multiple sections of the email. You can actually read the source code of an email. It is quite easy to follow and you will identify the different sections.

imap_fetchbody($inbox,$email_number,$section); 

http://php.net/manual/en/function.imap-fetchbody.php

(empty) - Entire message 0 - Message header 1 - MULTIPART/ALTERNATIVE 1.1 - TEXT/PLAIN 1.2 - TEXT/HTML 2 - file.ext 

The section you mention on the comments it is a PNG file. Some sections can be encoded in base64 as the email it is a text based protocol.

https://www.base64decode.org/

Example of an email:

 From: Nathaniel Borenstein <nsb@bellcore.com>   To:  Ned Freed <ned@innosoft.com>   Subject: Sample message   MIME-Version: 1.0   Content-type: multipart/mixed; boundary="simple   boundary"    This is the preamble.  It is to be ignored, though it   is a handy place for mail composers to include an   explanatory note to non-MIME compliant readers.   --simple boundary    This is implicitly typed plain ASCII text.   It does NOT end with a linebreak.   --simple boundary   Content-type: text/plain; charset=us-ascii    This is explicitly typed plain ASCII text.   It DOES end with a linebreak.    --simple boundary--   This is the epilogue.  It is also to be ignored. 

https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html

Read More

SQL Server: Record size larger than expected

Leave a Comment

My table consists of 3 columns

| Column Name | Data Type | Size | Value       | real      | 4 | LogId       | int       | 4 | SigId       | smallint  | 2 

One primary key is set for columns LogId, SigId.

The sum of all size's is 4+4+2=10, however using sys.dm_db_index_physical_statsI get, that the average (and min/max) record size in bytes is 25. Can someone explain? Am I comparing apples and oranges?

1 Answers

Answers 1

The physical record length includes row overhead in addition to the space needed for the actual column values. On my SQL Server instance, I get an average record length of 17 reported with the following table:

CREATE TABLE dbo.Example1(       Value real NOT NULL     , LogId int NOT NULL     , SigId smallint NOT NULL     , CONSTRAINT PK_Example1 PRIMARY KEY CLUSTERED(LogId, SigId) ); GO INSERT INTO dbo.Example1 (Value, LogId, SigId) VALUES(1, 2, 3); GO SELECT avg_record_size_in_bytes FROM sys.dm_db_index_physical_stats(DB_ID(), OBJECT_ID(N'dbo.Example1'),1,0,'DETAILED') WHERE index_level = 0; GO 

The 17 byte record length reported by sys.dm_db_index_physical_stats includes 10 bytes for data, 4 bytes for the record header, 2 bytes for the column count, and 1 byte for the NULL bitmap. See Paul Randal's Anatomy of a record article for details of the record structure.

Below is a script to dump the first clustered index data page using DBCC_PAGE as determined by the undocumented (don't use it in production) sys.dm_db_database_page_allocations table-valued function:

DECLARE       @database_id int = DB_ID()     , @object_id int = OBJECT_ID(N'dbo.Example1')     , @allocated_page_file_id int     , @allocated_page_page_id int; --get first clustered index data page SELECT       @allocated_page_file_id = allocated_page_file_id     , @allocated_page_page_id = allocated_page_page_id FROM sys.dm_db_database_page_allocations(@database_id, @object_id, 1, 1, 'DETAILED') WHERE     page_type_desc = N'DATA_PAGE'     AND previous_page_page_id IS NULL --first page of clustered index; --dump record DBCC TRACEON(3604); DBCC PAGE(@database_id,@allocated_page_file_id,@allocated_page_page_id,1); DBCC TRACEOFF(3604); GO 

Here is an excerpt from the results on my instance with the physical record structure fields called out:

DATA:   Slot 0, Offset 0x60, Length 17, DumpStyle BYTE  Record Type = PRIMARY_RECORD        Record Attributes =  NULL_BITMAP    Record Size = 17  Memory Dump @0x0000002262C7A060  0000000000000000:   10000e00 02000000 03000000 803f0300 00        .............?...                     |        |        |   |        |    |null bitmap (1 byte)                     |        |        |   |        |column count (2 bytes)                     |        |        |   |Value column data (4-byte real)                     |        |        |SigId column data (2-byte smallint)                     |        |LogId column data (4-byte int)                     |Record header (2-byte record type and 2 byte offset to null bitmap) 

As to why your actual record length is 25 instead of 17 as in this example, the likely cause is schema changes were made after the table was initially created as Martin suggested in his comment. If the database has a row-versioning isolation level enabled, there will be additional overhead as mentioned in Paul's blog post but I doubt that is the reason here since that overhead would be more than 8 bytes.

Read More

Friday, December 29, 2017

How to use TypeScript with Vue.js and Single File Components?

Leave a Comment

I have searched all over the web for a minimal, working example of a Vue.js + TypeScript setup. As per usual with the "modern JavaScript stack", most of these tutorials are either out-of-date despite being written just a couple of months ago or depending on a very specific setup. There appears to be no generic, verifiable example to build on.

Here are some of the resources I considered:

The basic template I use is the one provided by running vue-cli init webpack with all default options. As this produces a lot of code, I'm not pasting everything here. If there is need for some specific excerpts, I will gladly update the question.

The official Vue.js documentation is useless for my purpose because it doesn't consider setting up TypeScript with SFCs. The latest I tried was the last on of the list. I followed the setup precisely but it runs me into the following error on npm run dev:

[tsl] ERROR in /Users/[REDACTED]/ts-test/src/main.ts(12,3)       TS2345: Argument of type '{ el: string; router: any; template: string; components: { App: { name: string; }; }; }' is not assignable to parameter of type 'ComponentOptions<Vue, DefaultData<Vue>, DefaultMethods<Vue>, DefaultComputed, PropsDefinition<Rec...'.   Object literal may only specify known properties, and 'router' does not exist in type 'ComponentOptions<Vue, DefaultData<Vue>, DefaultMethods<Vue>, DefaultComputed, PropsDefinition<Rec...'. 

Can anyone shine some light on why this happens and how to resolve it? Better yet, I'd very much welcome a concise, minimal, step-by-step example of how to set up a working Vue.js + TypeScript configuration with the webpack template.

I have already successfully completed several client projects that run in production in Vue.js with vanilla JavaScript but this TypeScript tooling in combination with Vue.js just confuses the hell out of me.

3 Answers

Answers 1

I have tried to use typescript with vue. My personal opinion: it does not work well. Since some vue internals are not suited for typescript:

  1. vuex with this.$store.dispatch('some_action')
  2. Vue.use, Vue.mixin and other similar things that mutate the global Vue instance

But, while doing my research I have found these wonderful boilerplates: typescript-vue-tutorial by Daniel Rosenwasser and TypeScript-Vue-Starter by Microsoft.

I have also tried to mimic vue-webpack-template with typescript by myself: https://github.com/sobolevn/wemake-vue-template

There are also nice tools to make your typescript + vue workflow better:

In the end I have decided to use flow. Check this project template if you are interested.

Answers 2

Absolute agree @sobolevn's opinion. But, there is a lot of information for me to judge, community support for TypeScript is worth the wait.

Vue's ecosystem is being more TypeScript:

  1. *.vue file's TypeScript lint in VSCode be supported. Look this issue vetur.

  2. Open Source Ecosystem.

  3. New vue-cli in design. click me

So if you have time to wait, you can temporarily observe for some time. Or, you can refer to these projects: TypeScript-Vue-Starter and A Webpack Template Fork With Typescript Support.

Answers 3

This looks like the latest vue-cli template with a significant amount of stars and support for vue 2.5.

I don't think I saw it specifically mentioned in the other answers

vue init ducksoupdev/vue-webpack-typescript my-project

https://github.com/ducksoupdev/vue-webpack-typescript

Read More

How do I make JavaFX's webview render even when not visible due to being in another tab?

Leave a Comment

I'm loading a website on a JavaFX WebView and after a while taking a screenshot with something like:

WritableImage image = webView.snapshot(null, null); 

If I'm looking at that WebView that works fine, but if it's hidden by being in a tab that is not in the foreground (I'm not sure about other cases of hiding it), then, the screenshot is of the appropriate site, but entirely blank.

How can I force the WebView to render even if not visible?

During this time, webView.isVisible() is true.

I found there's a method in WebView called isTreeReallyVisible() and currently it contains:

private boolean isTreeReallyVisible() {     if (getScene() == null) {         return false;     }      final Window window = getScene().getWindow();      if (window == null) {         return false;     }      boolean iconified = (window instanceof Stage) ? ((Stage)window).isIconified() : false;      return impl_isTreeVisible()            && window.isShowing()            && window.getWidth() > 0            && window.getHeight() > 0            && !iconified; } 

When the WebView is hidden by being in a non-foreground tab, impl_isTreeVisible() is false (all other factors in the return statement are true). That method is on Node and looks like this:

/**  * @treatAsPrivate implementation detail  * @deprecated This is an internal API that is not intended for use and will be removed in the next version  */ @Deprecated public final boolean impl_isTreeVisible() {     return impl_treeVisibleProperty().get(); }  /**  * @treatAsPrivate implementation detail  * @deprecated This is an internal API that is not intended for use and will be removed in the next version  */ @Deprecated protected final BooleanExpression impl_treeVisibleProperty() {     if (treeVisibleRO == null) {         treeVisibleRO = new TreeVisiblePropertyReadOnly();     }     return treeVisibleRO; } 

I could have overriden impl_treeVisibleProperty() to provide my own implementation, but WebView is final, so, I cannot inherit from it.

Another completely different situation to being minimized (iconified) or on a hidden tab is to have the stage completely hidden (as in, running in the tray bar). When in that mode, even if I can get rendering to happen, the WebView doesn't resize. I call webView.resize() and then take a screenshot and the screenshot is of the appropriate size but the actual rendered page is of whatever size the WebView was before.

Debugging this sizing behavior in shown and hidden stages, I found that eventually we get to Node.addToSceneDirtyList() that contains:

private void addToSceneDirtyList() {     Scene s = getScene();     if (s != null) {         s.addToDirtyList(this);         if (getSubScene() != null) {             getSubScene().setDirty(this);         }     } } 

When in hidden mode, getScene() returns null, unlike what happens when it's being show. That means that s.addToDirtyList(this) is never called. I'm not sure if this is the reason why it doesn't get properly resized.

There's a bug about this, a very old one, here: https://bugs.openjdk.java.net/browse/JDK-8087569 but I don't think that's the whole issue.

I'm doing this with Java 1.8.0_151. I tried 9.0.1 to see if it would behave differently as it is my understanding that WebKit was upgraded, but no, it's the same.

3 Answers

Answers 1

Reproducing Pablo's problem here: https://github.com/johanwitters/stackoverflow-javafx-webview.

Pablo suggested to override WebView and adjust some methods. That doesn't work given it's a final class and a private member. As an alternative, I've used javassist to rename a method and replace the code with the code that I want it to execute. I've "replaced" the contents of method handleStagePulse, as shown below.

public class WebViewChanges { //    public static String MY_WEBVIEW_CLASSNAME = WebView.class.getName();      public WebView newWebView() {         createSubclass();         return new WebView();     }      // https://www.ibm.com/developerworks/library/j-dyn0916/index.html     boolean created = false;     private void createSubclass() {         if (created) return;         created = true;         try         {             String methodName = "handleStagePulse";              // get the super class             CtClass webViewClass = ClassPool.getDefault().get("javafx.scene.web.WebView");              // get the method you want to override             CtMethod handleStagePulseMethod = webViewClass.getDeclaredMethod(methodName);              // Rename the previous handleStagePulse method             String newName = methodName+"Old";             handleStagePulseMethod.setName(newName);              //  mnew.setBody(body.toString());             CtMethod newMethod = CtNewMethod.copy(handleStagePulseMethod, methodName, webViewClass, null);             String body = "{" +                     "  " + Scene.class.getName() + ".impl_setAllowPGAccess(true);\n" +                     "  " + "final " + NGWebView.class.getName() + " peer = impl_getPeer();\n" +                     "  " + "peer.update(); // creates new render queues\n" + //                    "  " + "if (page.isRepaintPending()) {\n" +                     "  " + "   impl_markDirty(" + DirtyBits.class.getName() + ".WEBVIEW_VIEW);\n" + //                    "  " + "}\n" +                     "  " + Scene.class.getName() + ".impl_setAllowPGAccess(false);\n" +                     "}\n";             System.out.println(body);             newMethod.setBody(body);             webViewClass.addMethod(newMethod);               CtMethod isTreeReallyVisibleMethod = webViewClass.getDeclaredMethod("isTreeReallyVisible");         }         catch (Exception e)         {             e.printStackTrace();         }     } } 

This snippet is called from the WebViewSample which opens 2 tabs. One with a "snapshot" button, another with the WebView. As Pablo pointed out, the tab with the WebView needs to be the second tab to be able to reproduce.

package com.johanw.stackoverflow;  import javafx.application.Application; import javafx.embed.swing.SwingFXUtils; import javafx.event.EventHandler; import javafx.geometry.HPos; import javafx.geometry.Pos; import javafx.geometry.VPos; import javafx.scene.Group; import javafx.scene.Node; import javafx.scene.Scene; import javafx.scene.control.Button; import javafx.scene.control.Label; import javafx.scene.control.Tab; import javafx.scene.control.TabPane; import javafx.scene.image.WritableImage; import javafx.scene.input.MouseEvent; import javafx.scene.layout.*; import javafx.scene.paint.Color; import javafx.scene.web.WebEngine; import javafx.scene.web.WebView; import javafx.stage.Stage;  import javax.imageio.ImageIO; import java.awt.image.RenderedImage; import java.io.File; import java.io.IOException;   public class WebViewSample extends Application {     private Scene scene;     private TheBrowser theBrowser;      private void setLabel(Label label) {         label.setText("" + theBrowser.browser.isVisible());     }      @Override     public void start(Stage primaryStage) {         primaryStage.setTitle("Tabs");         Group root = new Group();         Scene scene = new Scene(root, 400, 250, Color.WHITE);          TabPane tabPane = new TabPane();          BorderPane borderPane = new BorderPane();          theBrowser = new TheBrowser();         {             Tab tab = new Tab();             tab.setText("Other tab");              HBox hbox0 = new HBox();             {                 Button button = new Button("Screenshot");                 button.addEventHandler(MouseEvent.MOUSE_PRESSED,                         new EventHandler<MouseEvent>() {                             @Override                             public void handle(MouseEvent e) {                                 WritableImage image = theBrowser.getBrowser().snapshot(null, null);                                 File file = new File("test.png");                                 RenderedImage renderedImage = SwingFXUtils.fromFXImage(image, null);                                 try {                                     ImageIO.write(                                             renderedImage,                                             "png",                                             file);                                 } catch (IOException e1) {                                     e1.printStackTrace();                                 }                              }                         });                 hbox0.getChildren().add(button);                 hbox0.setAlignment(Pos.CENTER);             }              HBox hbox1 = new HBox();             Label visibleLabel = new Label("");             {                 hbox1.getChildren().add(new Label("webView.isVisible() = "));                 hbox1.getChildren().add(visibleLabel);                 hbox1.setAlignment(Pos.CENTER);                 setLabel(visibleLabel);             }             HBox hbox2 = new HBox();             {                 Button button = new Button("Refresh");                 button.addEventHandler(MouseEvent.MOUSE_PRESSED,                         new EventHandler<MouseEvent>() {                             @Override                             public void handle(MouseEvent e) {                                 setLabel(visibleLabel);                             }                         });                 hbox2.getChildren().add(button);                 hbox2.setAlignment(Pos.CENTER);             }             VBox vbox = new VBox();             vbox.getChildren().addAll(hbox0);             vbox.getChildren().addAll(hbox1);             vbox.getChildren().addAll(hbox2);             tab.setContent(vbox);             tabPane.getTabs().add(tab);         }         {             Tab tab = new Tab();             tab.setText("Browser tab");             HBox hbox = new HBox();             hbox.getChildren().add(theBrowser);             hbox.setAlignment(Pos.CENTER);             tab.setContent(hbox);             tabPane.getTabs().add(tab);         }          // bind to take available space         borderPane.prefHeightProperty().bind(scene.heightProperty());         borderPane.prefWidthProperty().bind(scene.widthProperty());          borderPane.setCenter(tabPane);         root.getChildren().add(borderPane);         primaryStage.setScene(scene);         primaryStage.show();     }      public static void main(String[] args){         launch(args);     } }  class TheBrowser extends Region {      final WebView browser;     final WebEngine webEngine;      public TheBrowser() {         browser = new WebViewChanges().newWebView();         webEngine = browser.getEngine();         getStyleClass().add("browser");         webEngine.load("http://www.google.com");         getChildren().add(browser);      }     private Node createSpacer() {         Region spacer = new Region();         HBox.setHgrow(spacer, Priority.ALWAYS);         return spacer;     }      @Override protected void layoutChildren() {         double w = getWidth();         double h = getHeight();         layoutInArea(browser,0,0,w,h,0, HPos.CENTER, VPos.CENTER);     }      @Override protected double computePrefWidth(double height) {         return 750;     }      @Override protected double computePrefHeight(double width) {         return 500;     }      public WebView getBrowser() {         return browser;     }      public WebEngine getWebEngine() {         return webEngine;     } } 

I've not succeeded in fixing Pablo's problem, but hopefully the suggestion to use javassist might help.

I'm sure: To be continued...

Answers 2

I would consider reloading page at the suitable moment using this command:

webView.getEngine().reload(); 

Also try to change parameter SnapshotParameters in method snapShot

If it would not work then I would consider storing Image in memory when WebView is being rendered on screen.

Answers 3

If you go by logical implementation of snapshot, only things that are visible on screen are taken as a snapshot. For taking snapshot of the web view, you can either make it automatically visible by clicking on the tab inside which the view is rendered just before taking the snapshot. Or you can manually click on the tab and take the screenshot. I think no API allows to take snapshot of hidden part as logically it will voilate the concept of hidden things. Code for taking snapshot is already available with you. You can click, or Load the tab like:

You can add a selectionChangedListener or you can do the load just before the snapshot.

addItemTab.setOnSelectionChanged(event -> loadTabBasedFXML(addItemTab, "/view/AddItem.fxml"));  private void loadTabBasedFXML(Tab tab, String fxmlPath) {         try {             AnchorPane anchorPane = FXMLLoader.load(this.getClass().getResource(fxmlPath));             tab.setContent(anchorPane);         } catch (IOException e) {         }     } 
Read More

How to build 1 jar from 3 sub-modules by using gradle

Leave a Comment

I have:

  • Android Studio 3
  • gradle 4.1
  • gradle tools 3: classpath 'com.android.tools.build:gradle:3.0.1'

When I had one module and used gradle tools 2, I used:

task makeJar(type: Copy) {     def releaseFolder = new File('build/intermediates/bundles/release/')     if (releaseFolder.exists()) {         from('build/intermediates/bundles/release/')     } else {         from('build/intermediates/bundles/default/')     }     into('build/outputs/jar/')     include('classes.jar')     rename ('classes.jar', 'MY-Android-SDK.jar') } 

Now have 3 modules:

              MainModule (com.my)                  /   \ (com.my.m1) Module1   Module2 (com.my.m2) 

I want to create MY-Android-SDK.jar file from all 3 modules.

So far I will be able to create 3 jar files per module and somehow merge them but I believe its not right way to do that.

Any ideas?

1 Answers

Answers 1

First of all, if you are creating android libraries, create AARs, not JARs. AAR files follow the android SDK structure and can have manifest files, and therefore, require permissions, and xml resources, among other things.

Second, if you are creating a framework from several different modules (a pretty reasonable approach in fact!), you should create different artifacts (AAR/JAR files). That makes easier to handle transitive dependencies, SDK level conflicts, consuming only what you need (there will be cases when you don't need half of your SDK and in that case discarding a few entries in the gradle files is way easier than fiddling with proguard to delete all the unused classes), providing alternative implementations or pluggable systems (for example, you create a contract module for, let's say, DFP, and a double click module. Then, by Inversion of Control, you fill the contract using double click, next month, they ask you to replace double click with amazon ads. You create a module with amazon ads following the contract module and replace the injected double click reference by amazon.)

Regardless of that, remember transitivity. Depending on how you bring your dependencies, everything can be added to the path by using a single dependency in the build.gradle file. If you use 'api':

api 'com.myname:moduleX:xxx' 

In module com.myname:moduleY, moduleX will be in the path as well. Use the Gradle maven plugin for creating the artifacts in the first place. That will allow you to create an android AAR file using the "install" task and will deploy it to your local maven repo (mavenLocal() in gradle). If you include all the submodules using "api", they will be added to the path. I don't recommend doing that, instead, import only what's necessary in each module and create separate AAR files for each module and consume them in your apps.

Now, if for some reason, you need to create a fat/Uber JAR, well, that process is called "shading". So, you'll need a shade plugin for gradle. Fortunately, there's a few. Just pick the best for your case (or use maven, and the maven shade plugin). Regardless, as someone with some experience developing libraries and frameworks, keep your modules as atomic and separate as possible. Lastly, if you need a working example for the gradle maven plugin, try this, it's old, but works as an example. good luck!

Read More

Thursday, December 28, 2017

Why are numpy functions so slow on pandas series / dataframes?

Leave a Comment

Consider a small MWE, taken from another question:

DateTime                Data 2017-11-21 18:54:31     1 2017-11-22 02:26:48     2 2017-11-22 10:19:44     3 2017-11-22 15:11:28     6 2017-11-22 23:21:58     7 2017-11-28 14:28:28    28 2017-11-28 14:36:40     0 2017-11-28 14:59:48     1 

The goal is to clip all values with an upper bound of 1. My answer uses np.clip, which works fine.

np.clip(df.Data, a_min=None, a_max=1) array([1, 1, 1, 1, 1, 1, 0, 1]) 

Or,

np.clip(df.Data.values, a_min=None, a_max=1) array([1, 1, 1, 1, 1, 1, 0, 1]) 

Both of which return the same answer. My question is about the relative performance of these two methods. Consider -

df = pd.concat([df]*1000).reset_index(drop=True)  %timeit np.clip(df.Data, a_min=None, a_max=1) 1000 loops, best of 3: 270 µs per loop  %timeit np.clip(df.Data.values, a_min=None, a_max=1) 10000 loops, best of 3: 23.4 µs per loop 

Why is there such a massive difference between the two, just by calling values on the latter? In other words...

Why are numpy functions so slow on pandas objects?

4 Answers

Answers 1

Yes, it seems like np.clip is a lot slower on pandas.Series than on numpy.ndarrays. That's correct but it's actually (at least asymptomatically) not that bad. 8000 elements is still in the regime where constant factors are major contributors in the runtime. I think this is a very important aspect to the question, so I'm visualizing this (borrowing from another answer):

# Setup  import pandas as pd import numpy as np  def on_series(s):     return np.clip(s, a_min=None, a_max=1)  def on_values_of_series(s):     return np.clip(s.values, a_min=None, a_max=1)  # Timing setup timings = {on_series: [], on_values_of_series: []} sizes = [2**i for i in range(1, 26, 2)]  # Timing for size in sizes:     func_input = pd.Series(np.random.randint(0, 30, size=size))     for func in timings:         res = %timeit -o func(func_input)         timings[func].append(res)  %matplotlib notebook  import matplotlib.pyplot as plt import numpy as np  fig, (ax1, ax2) = plt.subplots(1, 2)  for func in timings:     ax1.plot(sizes,               [time.best for time in timings[func]],               label=str(func.__name__)) ax1.set_xscale('log') ax1.set_yscale('log') ax1.set_xlabel('size') ax1.set_ylabel('time [seconds]') ax1.grid(which='both') ax1.legend()  baseline = on_values_of_series # choose one function as baseline for func in timings:     ax2.plot(sizes,               [time.best / ref.best for time, ref in zip(timings[func], timings[baseline])],               label=str(func.__name__)) ax2.set_yscale('log') ax2.set_xscale('log') ax2.set_xlabel('size') ax2.set_ylabel('time relative to {}'.format(baseline.__name__)) ax2.grid(which='both') ax2.legend()  plt.tight_layout() 

enter image description here

It's a log-log plot because I think this shows the important features more clearly. For example it shows that np.clip on a numpy.ndarray is faster but it also has a much smaller constant factor in that case. The difference for large arrays is only ~3! That's still a big difference but way less than the difference on small arrays.

However, that's still not an answer to the question where the time difference comes from.

The solution is actually quite simple: np.clip delegates to the clip method of the first argument:

>>> np.clip?? Source:    def clip(a, a_min, a_max, out=None):     """     ...     """     return _wrapfunc(a, 'clip', a_min, a_max, out=out)  >>> np.core.fromnumeric._wrapfunc?? Source:    def _wrapfunc(obj, method, *args, **kwds):     try:         return getattr(obj, method)(*args, **kwds)     # ...     except (AttributeError, TypeError):         return _wrapit(obj, method, *args, **kwds) 

The getattr line of the _wrapfunc function is the important line here, because np.ndarray.clip and pd.Series.clip are different methods, yes, completely different methods:

>>> np.ndarray.clip <method 'clip' of 'numpy.ndarray' objects> >>> pd.Series.clip <function pandas.core.generic.NDFrame.clip> 

Unfortunately is np.ndarray.clip a C-function so it's hard to profile it, however pd.Series.clip is a regular Python function so it's easy to profile. Let's use a Series of 5000 integers here:

s = pd.Series(np.random.randint(0, 100, 5000)) 

For the np.clip on the values I get the following line-profiling:

%load_ext line_profiler %lprun -f np.clip -f np.core.fromnumeric._wrapfunc np.clip(s.values, a_min=None, a_max=1)  Timer unit: 4.10256e-07 s  Total time: 2.25641e-05 s File: numpy\core\fromnumeric.py Function: clip at line 1673  Line #      Hits         Time  Per Hit   % Time  Line Contents ==============================================================   1673                                           def clip(a, a_min, a_max, out=None):   1674                                               """   ...   1726                                               """   1727         1           55     55.0    100.0      return _wrapfunc(a, 'clip', a_min, a_max, out=out)  Total time: 1.51795e-05 s File: numpy\core\fromnumeric.py Function: _wrapfunc at line 55  Line #      Hits         Time  Per Hit   % Time  Line Contents ==============================================================     55                                           def _wrapfunc(obj, method, *args, **kwds):     56         1            2      2.0      5.4      try:     57         1           35     35.0     94.6          return getattr(obj, method)(*args, **kwds)     58                                                59                                               # An AttributeError occurs if the object does not have     60                                               # such a method in its class.     61                                                62                                               # A TypeError occurs if the object does have such a method     63                                               # in its class, but its signature is not identical to that     64                                               # of NumPy's. This situation has occurred in the case of     65                                               # a downstream library like 'pandas'.     66                                               except (AttributeError, TypeError):     67                                                   return _wrapit(obj, method, *args, **kwds) 

But for np.clip on the Series I get a totally different profiling result:

%lprun -f np.clip -f np.core.fromnumeric._wrapfunc -f pd.Series.clip -f pd.Series._clip_with_scalar np.clip(s, a_min=None, a_max=1)  Timer unit: 4.10256e-07 s  Total time: 0.000823794 s File: numpy\core\fromnumeric.py Function: clip at line 1673  Line #      Hits         Time  Per Hit   % Time  Line Contents ==============================================================   1673                                           def clip(a, a_min, a_max, out=None):   1674                                               """   ...   1726                                               """   1727         1         2008   2008.0    100.0      return _wrapfunc(a, 'clip', a_min, a_max, out=out)  Total time: 0.00081846 s File: numpy\core\fromnumeric.py Function: _wrapfunc at line 55  Line #      Hits         Time  Per Hit   % Time  Line Contents ==============================================================     55                                           def _wrapfunc(obj, method, *args, **kwds):     56         1            2      2.0      0.1      try:     57         1         1993   1993.0     99.9          return getattr(obj, method)(*args, **kwds)     58                                                59                                               # An AttributeError occurs if the object does not have     60                                               # such a method in its class.     61                                                62                                               # A TypeError occurs if the object does have such a method     63                                               # in its class, but its signature is not identical to that     64                                               # of NumPy's. This situation has occurred in the case of     65                                               # a downstream library like 'pandas'.     66                                               except (AttributeError, TypeError):     67                                                   return _wrapit(obj, method, *args, **kwds)  Total time: 0.000804922 s File: pandas\core\generic.py Function: clip at line 4969  Line #      Hits         Time  Per Hit   % Time  Line Contents ==============================================================   4969                                               def clip(self, lower=None, upper=None, axis=None, inplace=False,   4970                                                        *args, **kwargs):   4971                                                   """   ...   5021                                                   """   5022         1           12     12.0      0.6          if isinstance(self, ABCPanel):   5023                                                       raise NotImplementedError("clip is not supported yet for panels")   5024                                              5025         1           10     10.0      0.5          inplace = validate_bool_kwarg(inplace, 'inplace')   5026                                              5027         1           69     69.0      3.5          axis = nv.validate_clip_with_axis(axis, args, kwargs)   5028                                              5029                                                   # GH 17276   5030                                                   # numpy doesn't like NaN as a clip value   5031                                                   # so ignore   5032         1          158    158.0      8.1          if np.any(pd.isnull(lower)):   5033         1            3      3.0      0.2              lower = None   5034         1           26     26.0      1.3          if np.any(pd.isnull(upper)):   5035                                                       upper = None   5036                                              5037                                                   # GH 2747 (arguments were reversed)   5038         1            1      1.0      0.1          if lower is not None and upper is not None:   5039                                                       if is_scalar(lower) and is_scalar(upper):   5040                                                           lower, upper = min(lower, upper), max(lower, upper)   5041                                              5042                                                   # fast-path for scalars   5043         1            1      1.0      0.1          if ((lower is None or (is_scalar(lower) and is_number(lower))) and   5044         1           28     28.0      1.4                  (upper is None or (is_scalar(upper) and is_number(upper)))):   5045         1         1654   1654.0     84.3              return self._clip_with_scalar(lower, upper, inplace=inplace)   5046                                              5047                                                   result = self   5048                                                   if lower is not None:   5049                                                       result = result.clip_lower(lower, axis, inplace=inplace)   5050                                                   if upper is not None:   5051                                                       if inplace:   5052                                                           result = self   5053                                                       result = result.clip_upper(upper, axis, inplace=inplace)   5054                                              5055                                                   return result  Total time: 0.000662153 s File: pandas\core\generic.py Function: _clip_with_scalar at line 4920  Line #      Hits         Time  Per Hit   % Time  Line Contents ==============================================================   4920                                               def _clip_with_scalar(self, lower, upper, inplace=False):   4921         1            2      2.0      0.1          if ((lower is not None and np.any(isna(lower))) or   4922         1           25     25.0      1.5                  (upper is not None and np.any(isna(upper)))):   4923                                                       raise ValueError("Cannot use an NA value as a clip threshold")   4924                                              4925         1           22     22.0      1.4          result = self.values   4926         1          571    571.0     35.4          mask = isna(result)   4927                                              4928         1           95     95.0      5.9          with np.errstate(all='ignore'):   4929         1            1      1.0      0.1              if upper is not None:   4930         1          141    141.0      8.7                  result = np.where(result >= upper, upper, result)   4931         1           33     33.0      2.0              if lower is not None:   4932                                                           result = np.where(result <= lower, lower, result)   4933         1           73     73.0      4.5          if np.any(mask):   4934                                                       result[mask] = np.nan   4935                                              4936         1           90     90.0      5.6          axes_dict = self._construct_axes_dict()   4937         1          558    558.0     34.6          result = self._constructor(result, **axes_dict).__finalize__(self)   4938                                              4939         1            2      2.0      0.1          if inplace:   4940                                                       self._update_inplace(result)   4941                                                   else:   4942         1            1      1.0      0.1              return result 

I stopped going into the subroutines at that point because it already highlights where the pd.Series.clip does much more work than the np.ndarray.clip. Just compare the total time of the np.clip call on the values (55 timer units) to one of the first checks in the pandas.Series.clip method, the if np.any(pd.isnull(lower)) (158 timer units). At that point the pandas method didn't even start at clipping and it already takes 3 times longer.

However several of these "overheads" become insignificant when the array is big:

s = pd.Series(np.random.randint(0, 100, 1000000))  %lprun -f np.clip -f np.core.fromnumeric._wrapfunc -f pd.Series.clip -f pd.Series._clip_with_scalar np.clip(s, a_min=None, a_max=1)  Timer unit: 4.10256e-07 s  Total time: 0.00593476 s File: numpy\core\fromnumeric.py Function: clip at line 1673  Line #      Hits         Time  Per Hit   % Time  Line Contents ==============================================================   1673                                           def clip(a, a_min, a_max, out=None):   1674                                               """   ...   1726                                               """   1727         1        14466  14466.0    100.0      return _wrapfunc(a, 'clip', a_min, a_max, out=out)  Total time: 0.00592779 s File: numpy\core\fromnumeric.py Function: _wrapfunc at line 55  Line #      Hits         Time  Per Hit   % Time  Line Contents ==============================================================     55                                           def _wrapfunc(obj, method, *args, **kwds):     56         1            1      1.0      0.0      try:     57         1        14448  14448.0    100.0          return getattr(obj, method)(*args, **kwds)     58                                                59                                               # An AttributeError occurs if the object does not have     60                                               # such a method in its class.     61                                                62                                               # A TypeError occurs if the object does have such a method     63                                               # in its class, but its signature is not identical to that     64                                               # of NumPy's. This situation has occurred in the case of     65                                               # a downstream library like 'pandas'.     66                                               except (AttributeError, TypeError):     67                                                   return _wrapit(obj, method, *args, **kwds)  Total time: 0.00591302 s File: pandas\core\generic.py Function: clip at line 4969  Line #      Hits         Time  Per Hit   % Time  Line Contents ==============================================================   4969                                               def clip(self, lower=None, upper=None, axis=None, inplace=False,   4970                                                        *args, **kwargs):   4971                                                   """   ...   5021                                                   """   5022         1           17     17.0      0.1          if isinstance(self, ABCPanel):   5023                                                       raise NotImplementedError("clip is not supported yet for panels")   5024                                              5025         1           14     14.0      0.1          inplace = validate_bool_kwarg(inplace, 'inplace')   5026                                              5027         1           97     97.0      0.7          axis = nv.validate_clip_with_axis(axis, args, kwargs)   5028                                              5029                                                   # GH 17276   5030                                                   # numpy doesn't like NaN as a clip value   5031                                                   # so ignore   5032         1          125    125.0      0.9          if np.any(pd.isnull(lower)):   5033         1            2      2.0      0.0              lower = None   5034         1           30     30.0      0.2          if np.any(pd.isnull(upper)):   5035                                                       upper = None   5036                                              5037                                                   # GH 2747 (arguments were reversed)   5038         1            2      2.0      0.0          if lower is not None and upper is not None:   5039                                                       if is_scalar(lower) and is_scalar(upper):   5040                                                           lower, upper = min(lower, upper), max(lower, upper)   5041                                              5042                                                   # fast-path for scalars   5043         1            2      2.0      0.0          if ((lower is None or (is_scalar(lower) and is_number(lower))) and   5044         1           32     32.0      0.2                  (upper is None or (is_scalar(upper) and is_number(upper)))):   5045         1        14092  14092.0     97.8              return self._clip_with_scalar(lower, upper, inplace=inplace)   5046                                              5047                                                   result = self   5048                                                   if lower is not None:   5049                                                       result = result.clip_lower(lower, axis, inplace=inplace)   5050                                                   if upper is not None:   5051                                                       if inplace:   5052                                                           result = self   5053                                                       result = result.clip_upper(upper, axis, inplace=inplace)   5054                                              5055                                                   return result  Total time: 0.00575753 s File: pandas\core\generic.py Function: _clip_with_scalar at line 4920  Line #      Hits         Time  Per Hit   % Time  Line Contents ==============================================================   4920                                               def _clip_with_scalar(self, lower, upper, inplace=False):   4921         1            2      2.0      0.0          if ((lower is not None and np.any(isna(lower))) or   4922         1           28     28.0      0.2                  (upper is not None and np.any(isna(upper)))):   4923                                                       raise ValueError("Cannot use an NA value as a clip threshold")   4924                                              4925         1          120    120.0      0.9          result = self.values   4926         1         3525   3525.0     25.1          mask = isna(result)   4927                                              4928         1           86     86.0      0.6          with np.errstate(all='ignore'):   4929         1            2      2.0      0.0              if upper is not None:   4930         1         9314   9314.0     66.4                  result = np.where(result >= upper, upper, result)   4931         1           61     61.0      0.4              if lower is not None:   4932                                                           result = np.where(result <= lower, lower, result)   4933         1          283    283.0      2.0          if np.any(mask):   4934                                                       result[mask] = np.nan   4935                                              4936         1           78     78.0      0.6          axes_dict = self._construct_axes_dict()   4937         1          532    532.0      3.8          result = self._constructor(result, **axes_dict).__finalize__(self)   4938                                              4939         1            2      2.0      0.0          if inplace:   4940                                                       self._update_inplace(result)   4941                                                   else:   4942         1            1      1.0      0.0              return result 

There are still multiple function calls, for example isna and np.where, that take a significant amount of time, but overall this is at least comparable to the np.ndarray.clip time (that's in the regime where the timing difference is ~3 on my computer).

The takeaway should probably be:

  • Many NumPy functions just delegate to a method of the object passed in, so there can be huge differences when you pass in different objects.
  • Profiling, especially line-profiling, can be a great tool to find the places where the performance difference comes from.
  • Always make sure to test differently sized objects in such cases. You could be comparing constant factors that probably don't matter except if you process lots of small arrays.

Used versions:

Python 3.6.3 64-bit on Windows 10 Numpy 1.13.3 Pandas 0.21.1 

Answers 2

Just read the source code, it's clear.

def clip(a, a_min, a_max, out=None):     """a : array_like Array containing elements to clip."""     return _wrapfunc(a, 'clip', a_min, a_max, out=out)  def _wrapfunc(obj, method, *args, **kwds):     try:         return getattr(obj, method)(*args, **kwds)     #This situation has occurred in the case of     # a downstream library like 'pandas'.     except (AttributeError, TypeError):         return _wrapit(obj, method, *args, **kwds)  def _wrapit(obj, method, *args, **kwds):     try:         wrap = obj.__array_wrap__     except AttributeError:         wrap = None     result = getattr(asarray(obj), method)(*args, **kwds)     if wrap:         if not isinstance(result, mu.ndarray):             result = asarray(result)         result = wrap(result)     return result 

rectify:

after pandas v0.13.0_ahl1,pandas has it's own implement of clip.

Answers 3

There are two parts to the performance difference to be aware of here:

  • Python overhead in each library (pandas being extra helpful)
  • Difference in numeric algorithm implementation (pd.clip actually calls np.where)

Running this on a very small array should demonstrate the difference in Python overhead. For numpy, this is understandably very small, however pandas does a lot of checking (null values, more flexible argument processing, etc) before getting to the heavy number crunching. I've tried to show a rough breakdown of the stages which the two codes go through before hitting C code bedrock.

data = pd.Series(np.random.random(100)) 

When using np.clip on an ndarray, the overhead is simply the numpy wrapper function calling the object's method:

>>> %timeit np.clip(data.values, 0.2, 0.8)        # numpy wrapper, calls .clip() on the ndarray >>> %timeit data.values.clip(0.2, 0.8)            # C function call  2.22 µs ± 125 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) 1.32 µs ± 20.4 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) 

Pandas spends more time checking for edge cases before getting to the algorithm:

>>> %timeit np.clip(data, a_min=0.2, a_max=0.8)   # numpy wrapper, calls .clip() on the Series >>> %timeit data.clip(lower=0.2, upper=0.8)       # pandas API method >>> %timeit data._clip_with_scalar(0.2, 0.8)      # lowest level python function  102 µs ± 1.54 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each) 90.4 µs ± 1.01 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each) 73.7 µs ± 805 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each) 

Relative to overall time, the overhead of both libraries before hitting C code is pretty significant. For numpy, the single wrapping instruction takes as much time to execute as the numeric processing. Pandas has ~30x more overhead just in the first two layers of function calls.

To isolate what is happening at the algorithm level, we should check this on a larger array and benchmark the same functions:

>>> data = pd.Series(np.random.random(1000000))  >>> %timeit np.clip(data.values, 0.2, 0.8) >>> %timeit data.values.clip(0.2, 0.8)  2.85 ms ± 37.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) 2.85 ms ± 15.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  >>> %timeit np.clip(data, a_min=0.2, a_max=0.8) >>> %timeit data.clip(lower=0.2, upper=0.8) >>> %timeit data._clip_with_scalar(0.2, 0.8)  12.3 ms ± 135 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) 12.3 ms ± 115 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) 12.2 ms ± 76.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) 

The python overhead in both cases is now negligible; time for wrapper functions and argument checking is small relative to the calculation time on 1 million values. However there is a 3-4x speed difference which can be attributed to numeric implementation. By investigating a bit in the source code, we see that the pandas implementation of clip actually uses np.where, not np.clip:

def clip_where(data, lower, upper):     ''' Actual implementation in pd.Series._clip_with_scalar (minus NaN handling). '''     result = data.values     result = np.where(result >= upper, upper, result)     result = np.where(result <= lower, lower, result)     return pd.Series(result)  def clip_clip(data, lower, upper):     ''' What would happen if we used ndarray.clip instead. '''     return pd.Series(data.values.clip(lower, upper)) 

The additional effort required to check each boolean condition separately before doing a conditional replace would seem to account for the speed difference. Specifying both upper and lower would result in 4 passes through the numpy array (two inequality checks and two calls to np.where). Benchmarking these two functions shows that 3-4x speed ratio:

>>> %timeit clip_clip(data, lower=0.2, upper=0.8) >>> %timeit clip_where(data, lower=0.2, upper=0.8)  11.1 ms ± 101 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) 2.97 ms ± 76.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) 

I'm not sure why the pandas devs went with this implementation. np.clip may be a newer API function that previously required a workaround. There is also a little more to it than I've gone into here, since pandas checks for various case before running the final algorithm, and this is only one of the implementations that may be called.

Answers 4

The reason why the performance differs is because numpy first tends to search for pandas implementation of the function using getattr than doing the same in builtin numpy functions when a pandas object is passed.

Its not the numpy over the pandas object that is slow, its the pandas version.

When you do

np.clip(pd.Series([1,2,3,4,5]),a_min=None,amax=1)   

_wrapfunc is being called :

# Code from source  def _wrapfunc(obj, method, *args, **kwds):     try:         return getattr(obj, method)(*args, **kwds) 

Due to _wrapfunc's getattr method :

getattr(pd.Series([1,2,3,4,5]),'clip')(None, 1) # Equivalent to `pd.Series([1,2,3,4,5]).clip(lower=None,upper=1)` # 0    1 # 1    1 # 2    1 # 3    1 # 4    1 # dtype: int64 

If you go through the pandas implementation there is a lot of pre checking work that is done. Its the reason why the functions which has the pandas implementation done via numpy has such difference in speed.

Not only clip, functions like cumsum,cumprod,reshape,searchsorted,transpose and much more uses pandas version of them than numpy when you pass them a pandas object.

It might appear numpy is doing the work over those objects but under the hood its the pandas function.

Read More