Wich is the most efficient way to iterate a directory?

Say I have a directory foo, with some number of subdirectories. Each of these subdirectories has between 0 and 5 files of variable length which I would like to process. My initial code looks like so:

    pool.query(`       SET SEARCH_PATH TO public,os_local;     `).then(() => fs.readdirSync(srcpath)         .filter(file => fs.lstatSync(path.join(srcpath, file)).isDirectory())         .map(dir => {           fs.access(`${srcpath + dir}/${dir}_Building.shp`, fs.constants.R_OK, (err) => {             if (!err) {               openShapeFile(`${srcpath + dir}/${dir}_Building.shp`).then((source) => source.read() .then(function dbWrite (result) {               if (result.done) {                 console.log(`done ${dir}`)               } else {     const query = `INSERT INTO os_local.buildings(geometry,                   id,                   featcode,                   version)                   VALUES(os_local.ST_GeomFromGeoJSON($1),                   $2,                   $3,                   $4) ON CONFLICT (id) DO UPDATE SET                     featcode=$3,                     geometry=os_local.ST_GeomFromGeoJSON($1),                     version=$4;`                 return pool.connect().then(client => {                   client.query(query, [geoJson.split('"[[').join('[[').split(']]"').join(']]'),                     result.value.properties.ID,                     result.value.properties.FEATCODE,                     version                   ]).then((result) => {                     return source.read().then(dbWrite)                   }).catch((err) => {                     console.log(err,                       query,                       geoJson.split('"[[').join('[[').split(']]"').join(']]'),                       result.value.properties.ID,                       result.value.properties.FEATCODE,                       version                     )                     return source.read().then(dbWrite)                   })                   client.release()                 })               }             })).catch(err => console.log('No Buildings', err))             }           })            fs.access(`${srcpath + dir}/${dir}__ImportantBuilding.shp`, fs.constants.R_OK, (err) => {             //read file one line at a time             //spin up connection in pg.pool, insert data           })            fs.access(`${srcpath + dir}/${dir}_Road.shp`, fs.constants.R_OK, (err) => {             //read file one line at a time             //spin up connection in pg.pool, insert data           })            fs.access(`${srcpath + dir}/${dir}_Glasshouse.shp`, fs.constants.R_OK, (err) => {             //read file one line at a time             //spin up connection in pg.pool, insert data           })            fs.access(`${srcpath + dir}/${dir}_RailwayStation.shp`, fs.constants.R_OK, (err) => {             //read file one line at a time             //spin up connection in pg.pool, insert data           })         }) 

This mostly works, but it ends up having to wait for the longest file to be fully processed in every subdirectory, resulting in practice in there always being only 1 connection to the database.

Is there a way I could rearchitect this to make better use of my computational resources, while limiting the number of active postgres connections and forcing code to wait until connections become available? (I set them to 20 in the pg poolConfig for node-postgres)

2 Answers

Answers 1

If you need to have your files processed in turn for a certain amount of time, then you can use Streams, timers(for scheduling) and process.nextTick(). There is great manual for understanding streams in nodejs.

Answers 2

Here is an example of getting directory contents using generators. You can start getting the first couple files right away and then use asynchronous code afterward to process files in parallel.

// Dependencies const fs = require('fs'); const path = require('path');  // The generator function (note the asterisk) function* getFilesInDirectory(fullPath, recursive = false) {     // Convert file names to full paths     let contents = fs.readdirSync(fullPath).map(file => {         return path.join(fullPath, file);     });      for(let i = 0; i < contents.length; i++) {         const childPath = contents[i];         let stats = fs.statSync(childPath);         if (stats.isFile()) {             yield childPath;         } else if (stats.isDirectory() && recursive) {             yield* getFilesInDirectory(childPath, true);         }     } } 

Usage:

function handleResults(results) {     ... // Returns a promise }  function processFile(file) {     ... // Returns a promise }  var files = getFilesInDirectory(__dirname, true); var result = files.next(); var promises = []; while(!result.done) {     console.log(result.value);     file = files.next();     // Process files in parallel     var promise = processFile(file).then(handleResults);     promises.push(promise); }  promise.all(promises).then() {     console.log(done); } 

Read More

Are there differences between .then(functionReference) and .then(function(value){return functionReference(value)})?

Given a named function utilized to handle a Promise value

function handlePromise(data) {   // do stuff with `data`   return data } 

a) Passing the named function handlePromise as a reference to .then()

promise.then(handlePromise) 

b) Using an anonymous or named function as parameter to .then() and returning the named function handlePromise with Promise value as parameter within the body of the anonymous or named function passed to .then()

promise.then(function /*[functionName]*/(data) {return handlePromise(data)}) 

Questions

1. Are there any differences between patterns a) and b)?

2. If the answer to 1. is yes, what are the differences that should be considered when using either pattern?

3 Answers

Answers 1

It is possible to create a case where there is a difference when no argument is passed, but it is a stretch and generally you should pass f and not function(x) { return f(x); } or x => f(x) because it is cleaner.

Here is an example causing a difference, the rationale is that functions that takes parameters can cause side effects with those parameters:

function f() {    if(arguments.length === 0) console.log("win");    else console.log("Hello World"); } const delay = ms => new Promise(r => setTimeout(r, ms)); // just a delay delay(500).then(f); // logs "Hello World"; delay(500).then(() => f()) // logs "win" 

Answers 2

There is no difference, function(x){return f(x)} === f.

For more info, you may want to read about eta-conversion in lambda calculus.

Answers 3

Logic

From a logic perspective there is not anything that would set them apart.

Source

From a source code and style perspective my personal taste is against inline function declarations as they are harder to read (when reading someone else's code, when reading my own its a work of art LOL)

Debugging

From a debugging perspective when the nesting gets deep its is harder to debug when you have a long stack trace of anonymous calls.

Performance

From a performance perspective it is browser dependent. With no significant difference using Firefox. Using Chrome Canary 60 and all versions before. Inline anonymous function declarations are significantly slower after the first call than defined function statements, and function expressions. This is true for both traditional and arrow functions.

Comparing the two alternatives and timing the while loop only

var i,j; const f = a => a; j = i = 10000;  while(i--) f(i);  // timed loop   while(j--) (a=>a)(j); // timed loop 

The pre defined function is executed 870% quicker than the inline function.

But I have yet to see anyone use promises in performance critical code, the difference in time on the test machine (win10 32bit) is 0.0018µs(*) for f(i) and 0.0157µs for (a=>a)(i)

_{(*) µs denotes microseconds 1/1,000,000th of a second}

Conclusion

The differences are small to insignificant, more a matter of personal taste and style than anything else. If you work in a team use the style outlined in their style guide, if you are project lead or work on your own, use what you are most comfortable with.

The edge case as shown in BenjaminGruenbaum answer I do not consider valid as he explicitly calls f() in then(()=>f()) without an argument. That is the same as const ff = () => f(); delay(0).then(ff) and not a quirk of how the function is defined.

Read More

HTML5 video - setting video.currentTime breaks the player

I am trying to interact with a 3rd-party html5 video player in Chrome. I am able to obtain a valid reference to it thusly:

document.getElementsByTagName("video")[1] 

...and the readyState is 4, so it's all good.

I can successfully (and with expected result) call:

document.getElementsByTagName("video")[1].play(); document.getElementsByTagName("video")[1].pause(); 

BUT when I call:

document.getElementsByTagName("video")[1].currentTime = 500; 

...the video freezes and it doesn't advance to the new currentTime. The video duration is much longer than 500 seconds, so it should be able to advance to that spot. I have tried other times besides 500, all with same result. If I examine currentTime, it is correct as to what I just set. But it doesn't actually go there. Also I can no longer interact with the video. It ignores any calls to play() or pause() after I try to set currentTime.

Before I call currentTime, when I call play() I get this valid promise back, and everything else still works:

After I call currentTime, when I call play(), I get this broken promise back, and now nothing works on that video object:

If you have a Hulu account you can easily observe this behavior on any video by simply trying it in the Chrome developer console.

3 Answers

Answers 1

Try below code, it will first pause then set your position then again play

document.getElementsByTagName("video")[1].pause(); document.getElementsByTagName("video")[1].currentTime = 500; document.getElementsByTagName("video")[1].play(); 

Answers 2

Why don't you try this code.

function setTime(tValue) {         //  if no video is loaded, this throws an exception              try {                 if (tValue == 0) {                     video.currentTime = tValue;                 }                 else {                     video.currentTime += tValue;                 }               } catch (err) {                  // errMessage(err) // show exception              errMessage("Video content might not be loaded");                }      } 

Answers 3

    var myVideo=document.getElementsByTagName("video")     if(myVideo[1] != undefind)         {              myVideo[1].currentTime=500;        }    /* or provide id to each video tag and use getElementById('id')    */     var myVideo=document.getElementById("videoId")     if(myVideo != undefind)         {              myVideo.currentTime=500;        } 

Read More

Use promise to process MySQL return value in node.js

I have a python background and is currently migrating to node.js. I have problem adjusting to node.js due to its asynchronous nature.

For example, I am trying to return a value from a MySQL function.

function getLastRecord(name) {     var connection = getMySQL_connection();      var query_str =     "SELECT name, " +     "FROM records " +        "WHERE (name = ?) " +     "LIMIT 1 ";      var query_var = [name];      var query = connection.query(query_str, query_var, function (err, rows, fields) {         //if (err) throw err;         if (err) {             //throw err;             console.log(err);             logger.info(err);         }         else {             //console.log(rows);             return rows;         }     }); //var query = connection.query(query_str, function (err, rows, fields) { }  var rows = getLastRecord('name_record');  console.log(rows); 

After some reading up, I realize the above code cannot work and I need to return a promise due to node.js's asynchronous nature. I cannot write node.js code like python. How do I convert getLastRecord() to return a promise and how do I handle the returned value?

In fact, what I want to do is something like this;

if (getLastRecord() > 20> {     console.log("action"); } 

How can this be done in node.js in a readable way?

I would like to see how promises can be implemented in this case using bluebird.

5 Answers

Answers 1

This is gonna be a little scattered, forgive me.

First, assuming this code uses the mysql driver API correctly, here's one way you could wrap it to work with a native promise:

function getLastRecord(name) {     return new Promise(function(resolve, reject) {         // If you use bluebird, it comes with a Promise.try helper         // for avoiding this. The *point* of this try is to make         // sure that all errors get handled by the promise chain.         // Functions using nodebacks have a concept of "unchecked"         // errors that get thrown rather than passed to the callback         // but this is considered Bad Form in promise-land         try {             var connection = getMySQL_connection();              var query_str =             "SELECT name, " +             "FROM records " +                "WHERE (name = ?) " +             "LIMIT 1 ";              var query_var = [name];              connection.query(query_str, query_var, function (err, rows, fields) {                 // Call reject on error states,                 // call resolve with results                 if (err) {                     return reject(err);                 }                 resolve(rows);             });         } catch (err) {             reject(err);         }     }); }  getLastRecord('name_record').then(function(rows) {     // now you have your rows, you can see if there are <20 of them }).catch((err) => setImmediate(() => { throw err; })); // Throw async to escape the promise chain 

So one thing: You still have callbacks. Callbacks are just functions that you hand to something to call at some point in the future with arguments of its choosing. So the function arguments in xs.map(fn), the (err, result) functions seen in node and the promise result and error handlers are all callbacks. This is somewhat confused by people referring to a specific kind of callback as "callbacks," the ones of (err, result) used in node core in what's called "continuation-passing style", sometimes called "nodebacks" by people that don't really like them.

For now, at least (async/await is coming eventually), you're pretty much stuck with callbacks, regardless of whether you adopt promises or not.

Also, I'll note that promises aren't immediately, obviously helpful here, as you still have a callback. Promises only really shine when you combine them with Promise.all and promise accumulators a la Array.prototype.reduce. But they do shine sometimes, and they are worth learning.

Answers 2

You don't need to use promises, you can use a callback function, something like that:

function getLastRecord(name, next) {     var connection = getMySQL_connection();      var query_str =     "SELECT name, " +     "FROM records " +         "LIMIT 1 ";      var query_var = [name];      var query = connection.query(query_str, query_var, function (err, rows, fields) {         //if (err) throw err;         if (err) {             //throw err;             console.log(err);             logger.info(err);             next(err);         }         else {             //console.log(rows);             next(null, rows);         }     }); //var query = connection.query(query_str, function (err, rows, fields) { }  getLastRecord('name_record', function(err, data) {    if(err) {       // handle the error    } else {       // handle your data     } }); 

Answers 3

I have modified your code to use Q(NPM module) promises. I Assumed your 'getLastRecord()' function that you specified in above snippet works correctly.

You can refer following link to get hold of Q module

Click here : Q documentation

var q = require('q');  function getLastRecord(name) {  var deferred = q.defer(); // Use Q  var connection = getMySQL_connection();  var query_str = "SELECT name, " + "FROM records " +    "WHERE (name = ?) " + "LIMIT 1 ";  var query_var = [name];  var query = connection.query(query_str, query_var, function (err, rows, fields) {     //if (err) throw err;     if (err) {         //throw err;                    deferred.reject(err);     }     else {         //console.log(rows);                    deferred.resolve(rows);     } }); //var query = connection.query(query_str, function (err, rows, fields) {  return deferred.promise; }    // Call the method like this getLastRecord('name_record')  .then(function(rows){    // This function get called, when success    console.log(rows);   },function(error){    // This function get called, when error    console.log(error);   }); 

Answers 4

To answer your initial question: How can this be done in node.js in a readable way?

There is a library called co, which gives you the possibility to write async code in a synchronous workflow. Just have a look and npm install co.

The problem you face very often with that approach, is, that you do not get Promise back from all the libraries you like to use. So you have either wrap it yourself (see answer from @Joshua Holbrook) or look for a wrapper (for example: npm install mysql-promise)

(Btw: its on the roadmap for ES7 to have native support for this type of workflow with the keywords async await, but its not yet in node: node feature list.)

Answers 5

This can be achieved quite simply, for example with bluebird, as you asked:

var Promise = require('bluebird');  function getLastRecord(name) {     return new Promise(function(resolve, reject){         var connection = getMySQL_connection();          var query_str =             "SELECT name, " +             "FROM records " +             "WHERE (name = ?) " +             "LIMIT 1 ";          var query_var = [name];          var query = connection.query(query_str, query_var, function (err, rows, fields) {             //if (err) throw err;             if (err) {                 //throw err;                 console.log(err);                 logger.info(err);                 reject(err);             }             else {                 resolve(rows);                 //console.log(rows);             }         }); //var query = connection.query(query_str, function (err, rows, fields) {     }); }   getLastRecord('name_record')     .then(function(rows){         if (rows > 20) {             console.log("action");         }     })     .error(function(e){console.log("Error handler " + e)})     .catch(function(e){console.log("Catch handler " + e)}); 

Read More

Axios interceptors and asynchronous login

I'm implementing token authentication in my web app. My access token expires every N minutes and than a refresh token is used to log in and get a new access token.

I use Axios for all my API calls. I have an interceptor set up to intercept 401 responses.

axios.interceptors.response.use(undefined, function (err) {   if (err.status === 401 && err.config && !err.config.__isRetryRequest) {     serviceRefreshLogin(       getRefreshToken(),       success => { setTokens(success.access_token, success.refresh_token) },       error => { console.log('Refresh login error: ', error) }     )     err.config.__isRetryRequest = true     err.config.headers.Authorization = 'Bearer ' + getAccessToken()     return axios(err.config);   }   throw err }) 

Basically, as I intercept a 401 response, I want to do a login and than retry the original rejected request with the new tokens. My serviceRefreshLogin function calls setAccessToken() in its then block. But the problem is that the then block happens later than the getAccessToken() in the interceptor, so the retry happens with the old expired credentials.

getAccessToken() and getRefreshToken() simply return the existing tokens stored in the browser (they manage localStorage, cookies, etc).

How would I go about ensuring statements do not execute until a promise returns?

1 Answers

Answers 1

Just use another promise :D

axios.interceptors.response.use(undefined, function (err) {     return new Promise(function (resolve, reject) {         if (err.status === 401 && err.config && !err.config.__isRetryRequest) {             serviceRefreshLogin(                 getRefreshToken(),                 success => {                          setTokens(success.access_token, success.refresh_token)                          err.config.__isRetryRequest = true                         err.config.headers.Authorization = 'Bearer ' + getAccessToken();                         axios(err.config).then(resolve, reject);                 },                 error => {                      console.log('Refresh login error: ', error);                     reject(error);                  }             );         }         throw err;     }); }); 

If your enviroment doesn't suport promises use polyfill, for example https://github.com/stefanpenner/es6-promise

But, it may be better to rewrite getRefreshToken to return promise and then make code simpler

axios.interceptors.response.use(undefined, function (err) {          if (err.status === 401 && err.config && !err.config.__isRetryRequest) {             return getRefreshToken()             .then(function (success) {                 setTokens(success.access_token, success.refresh_token) ;                                    err.config.__isRetryRequest = true;                 err.config.headers.Authorization = 'Bearer ' + getAccessToken();                 return axios(err.config);             })             .catch(function (error) {                 console.log('Refresh login error: ', error);                 throw error;             });         }         throw err; }); 

Read More

How to send result of an yield on a promise as a stream in express 4?

Is there a way i could send the result of yielding a promise as a steam?

The json payload in some cases will be huge, it would make sense sending it as a stream.

function aPromise() {     // result of a postgres query. Using `pg`     return Promise.resolve({         'data': [1, 2 ,3]     }   }); }  //express 4 router let wrap = require('co-express');  router.get('/', wrap(function* (req, res, next) {     let payload;      try {         payload = yield aPromise();     } catch (e) {         return next(e);     }      res.json(payload); }));  

1 Answers

Answers 1

If you're fetching and loading into memory the whole object of your response, it's going to be hard to then reasonably turn it into streaming again. Anyway, streaming JSON implementation is needed here.

You'd have to share more details about how you're fetching the data. The choice of reasonable solution depends on this entirely.

Here's two examples of how you can make it work, where I made some assumptions:

1. You are capable of creating a stream of tuples like this: [key, data] for each key you want to put in the resulting json being sent out. This assumes your data is a lot of keys.

   res.set('content-type', 'application/json'); keystream.pipe(JSONStream.stringifyObject()).pipe(res)

2. You are capable of creating a stream of objects that you want to return as items in an array for the final json response.

   res.set('content-type', 'application/json'); itemsstream.pipe(JSONStream.stringify()).pipe(res)

Read More

TypeError: Cannot call method 'then' of undefined Angularjs

I am pretty new to Angular and have problems with making a synchronous operation. I have resolved few issues which came my way with the angular controller, where I get the error 'Cannot call method then of undefined' thrown from the newController file.

angular.module('newApp.newController', ['angularSpinner', 'ui.bootstrap'])  .controller('newController', function($q, $scope, utilityFactory, $http) {     utilityFactory.getData().then(function(data) {          console.log("success");         console.log(data);      }); });   angular.module('newApp.utility', [])     .factory('utilityFactory', function($q, $http) {          var utils = {};          //This is a cordova plugin         var getLauncher = function() {             return window.plugin.launcher;         };          var success = function(data) {             console.log(device);             return device;         }         var fail = function(error) {             console.log("error", error);         };          utils.getData = function() {             /* Get the store number details initially before initalizing the application */             if (window.plugin) {                 var launcher = getLauncher();                 console.log("Fetching data from device");                 //Cordova js is returning this method                 return launcher.getDevice(success, fail);             }         };         return utils;     }) 

3 Answers

Answers 1

With the understanding that :

Launcher.prototype.getDevice = function(successCallback, failureCallback) {     exec(successCallback, failureCallback, KEY, 'getDevice', []); } 

, we know that window.plugin.launcher.getDevice() returns undefined, not a data object. Instead, it provides its response via its success/failure callbacks (like nodebacks but with the args reversed).

Therefore, to work with promises, window.plugin.launcher.getDevice() needs to be "promisified", involving the explicit creation of a new Promise() and its resolution/rejection by .getDevice's callbacks. (Simply wrapping in $q(...) is not the same thing, and won't work).

angular.module('newApp.utility', []).factory('utilityFactory', function($q, $http) {     return {         getDevice: function() {             return $q.defer(function(resolve, reject) {                 window.plugin.launcher.getDevice(resolve, reject); // If this line throws for whatever reason, it will be automatically caught internally by Promise, and `reject(error)` will be called. Therefore you needn't explicitly fork for cases where `window.plugin` or `window.plugin.launcher` doesn't exist.             }).promise;         }     }; }); 

Calling from the controller should now work :

angular.module('newApp.newController', ['angularSpinner', 'ui.bootstrap']).controller('newController', function($q, $scope, utilityFactory, $http) {     return utilityFactory.getDevice().then(function(data) {         console.log(data);     }).catch(function(error) {         console.error(error);     }); }); 

Answers 2

    return launcher.getDevice(success, fail); 

this line is the issue, I would just wrap it with a promise:

    return $q(launcher.getDevice.bind(launcher, success, fail)); 

Edit: also you need to take care of else condition, so code would be:

    utils.getData = function() {         /* Get the store number details initially before initalizing the application */         if (window.plugin) {             var launcher = getLauncher();             console.log("Fetching data from device");             //Cordova js is returning this method             return $q(launcher.getDevice.bind(launcher, success, fail));         }         return $q.resolve(); // or $q.reject(reason);     }; 

Answers 3

1) Your actual Module should be "newApp", not "newApp.newController" and 'newApp.utility'. That is placing those two components into separate modules instead of in the myApp module.

2) You should only use the syntax of

angular.module('newApp', []) 

whenever you are declaring a new module. When you want to access the module, you should use

angular.module('newApp') 

https://docs.angularjs.org/api/ng/function/angular.module

3) Your utilityFactory is returning a variable 'device' that hasn't been declared anywhere

4) You can't use 'then' without returning a promise in your getData function. Then is a method that is implemented in Javascript promises, so you can't just use it anywhere in your code. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/then

utils.getData = function() {     var deferred = $q.defer();      if (window.plugin) {         var launcher = getLauncher();         console.log("Fetching data from device");         //Cordova js is returning this method         return launcher.getDevice(success, fail);     }      return deferred.promise;     }; 

Here is a codepen I used when debugging your code. I modified your code a little, but it will give an example of the function working when returning a promise. http://codepen.io/anon/pen/QNEEyx?editors=1010

Read More