R nested foreach %dopar% in outer loop and %do% in inner loop

I'm running the following script in R. If I use a %do% rather than a %dopar% the script works fine. However, if in the outer loop I use a %dopar% the loop runs forever without throwing any error (constant increase in memory usage until it goes out of memory). I'm using 16 cores.

library(parallel) library(foreach) library(doSNOW) library(dplyr)   NumberOfCluster <- 16  cl <- makeCluster(NumberOfCluster)  registerDoSNOW(cl)    foreach(i = UNSPSC_list, .packages = c('data.table', 'dplyr'), .verbose = TRUE) %dopar%      {        terms <- as.data.table(unique(gsub(" ", "", unlist(terms_list_by_UNSPSC$Terms[which(substr(terms_list_by_UNSPSC$UNSPSC,1,6) == i)]))))        temp <- inner_join(N_of_UNSPSCs_by_Term, terms, on = 'V1')        temp$V2 <- 1/as.numeric(temp$V2)       temp <- temp[order(temp$V2, decreasing = TRUE),]       names(temp) <- c('Term','Imp')       ABNs <- unique(UNSPSCs_per_ABN[which(substr(UNSPSCs_per_ABN$UNSPSC,1,4) == substr(i,1,4)), 1])        predictions <- as.numeric(vector())        predictions <- foreach (j = seq(1 : nrow(train)), .combine = 'c', .packages = 'dplyr')  %do%        {          descr <- names(which(!is.na(train[j,]) == TRUE))          if(unlist(predict_all[j,1]) %in% unlist(ABNs) || !unlist(predict_all[j,1]) %in% unlist(suppliers)) {union_all(predictions, sum(temp$Imp[which(temp$Term %in% descr)]))} else {union_all(predictions, 0)}            }      save(predictions, file = paste("Predictions", i,".rda", sep = "_"))      } 

1 Answers

Answers 1

The proper way of nesting foreach loop is using %:% operator. See the example. I have tested it on Windows.

library(foreach) library(doSNOW)  NumberOfCluster <- 4 cl <- makeCluster(NumberOfCluster)  registerDoSNOW(cl)   N <- 1e6  system.time(foreach(i = 1:10, .combine = rbind) %:%               foreach(j = 1:10, .combine = c) %do% mean(rnorm(N, i, j)))  system.time(foreach(i = 1:10, .combine = rbind) %:%               foreach(j = 1:10, .combine = c) %dopar% mean(rnorm(N, i, j))) 

Output:

> system.time(foreach(i = 1:10, .combine = rbind) %:% +               foreach(j = 1:10, .combine = c) %do% mean(rnorm(N, i, j)))    user  system elapsed     7.38    0.23    7.64  > system.time(foreach(i = 1:10, .combine = rbind) %:% +               foreach(j = 1:10, .combine = c) %dopar% mean(rnorm(N, i, j)))    user  system elapsed     0.09    0.00    2.14  

Read More

Using source() within parallel foreach loops

Here is a toy example to illustrate my problem.

library(foreach) library(doMC) registerDoMC(cores=2)  foreach(i = 1:2) %dopar%{   i + 2 } [[1]] [1] 3  [[2]] [1] 4 

So far so good...

But if the code i + 2 is saved in the file addition.R and that I call that file using source() then

> foreach(i = 1:2) %dopar%{ +   source("addition.R") + } Error in { : task 1 failed - "object 'i' not found" 

3 Answers

Answers 1

I cannot fully reproduce your toy, but I had a smiliar problem, which I was able to solve by:

source(file, local = TRUE) 

which should parse the source in the local environment, i.e. recognizing i.

Answers 2

I finally solved the problem by converting the source("addition.R") to a function and simply passing the variables into it. I don't know why but the suggested solutions based on source(file, local = TRUE) does not work.

Answers 3

The comment by NiceE and the answer by Sosel already address this; when calling source(file) it defaults to source(file, local = FALSE), which means that the code in the file sourced is evaluating in the global environment ("user's workspace") and there is, cf. ?source. Note that there is no variable i in the global environment. The solution is to make sure the file sourced in the environment that calls it, i.e. to use source(file, local = TRUE).

Solution:

library("foreach")  y <- foreach(i = 1:2) %dopar% {   i + 2 } str(y)  doMC::registerDoMC(cores = 2L) y <- foreach(i = 1:2) %dopar% {   source("addition.R", local = TRUE) } str(y) 

Example of the same problem with a for() loop:

The fact that source() is evaluated in the global environment which is different from the calling environment where i lives can also be illustrated using a regular for loop by running the for loop in another environment than the global, e.g. inside a function or by:

local({   for(i in 1:2) {     source("addition.R")   } }) 

which gives:

Error in eval(ei, envir) : object 'i' not found 

Now, the reason why the above foreach(i = 1:2) %dopar% { source("addition.R") } works with registerDoSEQ() if and only if called from the global environment, is that then the foreach iteration is evaluated in the calling environment, which is the global environment, which is the environment that source() uses. However, if one used local(foreach(i = 1:2) %dopar% { ... }) also this fails analoguously to the above local(for(i in 1:2) { ... }) call.

In conclusion: nothing magic happens, but to understand it is a bit tedious.

Read More

Chrome for loop optimization

So I was curious what would be faster for iterating through an array, the normal for loop or forEach so I executed this code in the console:

var arr = []; arr.length = 10000000; //arr.fill(1); for (var i_1 = 0; i_1 < arr.length; i_1++) { arr[i_1] = 1; } ////////////////////////////////// var t = new Date(); var sum = 0; for (var i = 0; i < arr.length; i++) {     var a = arr[i];     if (a & 1) {         sum += a;     }     else {         sum -= a;     } } console.log(new Date().getTime() - t.getTime()); console.log(sum);  t = new Date(); sum = 0; arr.forEach(function (value, i, aray) {     var a = value;     if (a & 1) {         sum += a;     }     else {         sum -= a;     } }); console.log(new Date().getTime() - t.getTime()); console.log(sum); 

Now the results in Chrome are 49ms for the for loop, 376ms for the forEach loop. Which is ok but the results in Firefox and IE (and Edge) are a lot different.

In both other browsers the first loop takes ~15 seconds (yes seconds) while the forEach takes "only" ~4 seconds.

My question is can someone tell me the exact reason Chrome is so much faster?

I tried all kinds of operations inside the loops, the results were always in favor for Chrome by a mile.

2 Answers

Answers 1

Disclaimer: I do not know the specifics of V8 in Chrome or the interpreter of Firefox / Edge, but there are some very general insights. Since V8 compiles Javascript to native code, let's see what it potentially could do:

• Very crudely: variables like your var i can be modelled as a very general Javascript variable, so that it can take any type of value from numbers to objects (modelled as a pointer to a struct Variable for instance), or the compiler can deduce the actual type (say an int in C++ for instance) from your JS and compile it like that. The latter uses less memory, exploits caching, uses less indirection, and can potentially be as fast as a for-loop in C++. V8 probably does this.
• The above holds for your array as well: maybe it compiles to a memory efficient array of ints stored contiguously in memory; maybe it is an array of pointers to general objects.
• Temporary variables can be removed.
• The second loop could be optimized by inlining the function call, maybe this is done, maybe it isn't.

The point being: all JS interpreters / compilers can potentially exploit these optimizations. This depends on a lot of factors: the trade-off between compilation and execution time, the way JS is written, etc.

V8 seems to optimize a lot, Firefox / Edge maybe don't in this example. Knowing why precisely requires in-depth understanding of the interpreter / compiler.

Answers 2

For loop is the afastest when compared to other iterators in every browser. But when comparing browsers ie is the slowest in iteration of for loops. Go and try jsperf.com for optimization is going to be my best recommendation. V8 engine implementation is the reason. After chrome split from webkit it stripped off more than 10k line of code in first few days.

Read More

httpUploadProgress is not working as expected for Buffer data?

I am new to node js/express .I am trying to make multiple image uploading app with cloudfront and s3 bucket.And I want to show progress bar for user I am using socket io for that.Photo uploading progress will be in loop.but the problem is when photo uploading starts it is always show 100% completion not from the beginning.Don't think the upload is completed it is not, My file is 20MB .I don't what's happening did i miss Something

this is my code

app.post('/posttodb',(req,res) => {     let isLoggedIn = req.cookies['check'];    let token = req.cookies['peace'];     const bucketName = 'awsBucketName';     console.log(isLoggedIn);     if(isLoggedIn == "true"){         if(token){             jwt.verify(token,JWTPASS,(err,decode) => {                  if(err){                     console.log(err)                     res.json({error:true})                 }else{                     console.log('========================>',decode)                     let postOwneranme = decode.user.username;                     let postTags = req.body.data.postTags;                     let photosBlob = req.body.data.photos;                     let postId = req.body.data.postId;                     let nepostAwsPhots = [];                     let OwnerPic = decode.user.propic;                     let postOwnerFullName = decode.user.name;                     let isMature = req.body.data.isMature;                     let postThumbUrl = req.body.data.thumnailUrl;                     let time = new Date();                      let tagSlug  = req.body.data.tagSlug;                       function savetodb() {                         console.log('Inserting all into DB');                         r.connect({db:'image'}).then(conn => {                              r.table('posts').insert({postId:postId,username:postOwneranme,tag:postTags,postUrlsAndCaptions:nepostAwsPhots,comments:[],postOwnerPic:OwnerPic,likesCount:0,whoLikedIt:[],views:0,postedTime:time,postOwnerFullName:postOwnerFullName,isMature:isMature,thumNailUrl:postThumbUrl,tagSlug:tagSlug}).run(conn).then(response => {                                 console.log(response)                                  if(response.inserted > 0){                                     console.log('Done Bro')                                     res.json({okva:true,postId:postId,username:postOwneranme})                                 }else{                                     res.json({okva:false})                                 }                             })                         })                        }                     function seemsToHaveNetworkProblem() {                         res.json({okva:false,message:"Seems To Have Network Problem"})                     }                           forEachOf(photosBlob,(value,key,callback) => {                             console.log(value.id);                             let newImageUriWillBe = value.blobData;                             let newImageNamewillBe = value.id;                             let imageType = value.ImageType;                             let caption = value.caption;                             console.log(imageType)                             let buf = new Buffer(newImageUriWillBe.replace(/^data:image\/\w+;base64,/, ""),'base64');                             s3.createBucket({Bucket:bucketName},() => {                                 let params = {Bucket: bucketName, Key: postOwneranme+'/'+newImageNamewillBe, Body: buf,ContentType:imageType,ContentLength:buf.length,ACL:'public-read'};                                 s3.upload(params,(err,data) => {                                     if(err){                                         callback(err);                                     }else{                                         // console.log("Successfully uploaded data to " + bucketName + "/" + id);                                         // console.log(`https://s3.amazonaws.com/${bucketName}/${username}/${id}`);                                         let response = {                                             picUrl:`https://s3.amazonaws.com/${bucketName}/${postOwneranme}/${newImageNamewillBe}`,                                             cation:caption                                          }                                           nepostAwsPhots.push(response);                                           callback()                                          // console.log(nepostAwsPhots)                                         res.writeHead(200, {'content-type': 'text/plain'});                                         res.end('Ok');                                        }                                 })                                //Problem Comes here                                      .on('httpUploadProgress', function(evt) {                                      let per = Math.round((evt.loaded * 100) / evt.total)                                     console.log('Progress:',per);                                      Socket.emit('Scoket',{proccesing:per})                                        })                               });                      },(err) => {                         if(err){                             console.log("From Node Loop error",err);                             seemsToHaveNetworkProblem()                         }else {                             savetodb()                         }                     })                   }             })         }else {             res.json({error:true});             console.log('OMG')         }     }else{         res.json({error:true});         console.log('OMG')     }      }); 

2 Answers

Answers 1

first check if you have updated library to the latest version and Try Modifying your code with this one:

.on('httpUploadProgress',function(progress) {     console.log(Math.round(progress.loaded/progress.total*100)+ '% done');     }); 

Answers 2

I am not exactly sure what is going on, but i think this one helps with your problem. Although, the variables used in this are from another similar source. Try adopting the method.

ss(socket).emit('strimage', stream, {size: file.size,name: fileName,email: emailid}); //initialize var to 0                     var blobStream = ss.createBlobReadStream(file);                     var size = 0;                     var uploadedSize;                     blobStream.on('data', function(chunk) {                       size += chunk.length; //try giving an upload size                         uploadedSize = Math.floor(size / file.size * 100)                       console.log(uploadedSize + '%');                         if (uploadedSize == 100) {                             console.log("inside uploadedSize");                             socket.emit('uploadcomplete', data);                         }                     });                      blobStream.pipe(stream); 

Read More

Foreach is picking the first checkbox only if checked

I am working on the following code in order to pick checkboxes from a form. If i check the first checkbox everything works great. If i check another checkbox i get the "Undefined index" error. Keep in mind that i am getting the checkboxes with post method and the submit button is above the checkboxes due to the complexity of the location of the form and the fields. What i need essentially is to pick multiple checkboxes and add certain values to the database.

<?php    session_start();   if($_SESSION['admin_logged_in'] != true){     header("Location:login.html");     exit();   }   include 'db.php';    $from = mysql_real_escape_string($_GET['from']);   $room = mysql_real_escape_string($_POST['room']);    if(!empty($_POST['id'])) {     foreach($_POST['id'] as $check) {       $id = $check;        $sel = mysql_query("select * from $from where id = '$id' limit 1 ") or die(mysql_error());        while($row = mysql_fetch_array($sel)){         $preview = $row['preview'];         $text = $row['text'];         $title = $row['title'];         $images = $row['images'];       }        $ins = mysql_query("insert into $room (id, preview, text, title, images) values (' ', '$preview', '$text', '$title', '$images') ") or die(mysql_error());      }      header("Location:admin.php");   }  ?> 

The code of the form can be found below:

<form class="form-inline" name="bulkcopy" method="post" action="bulkcopy.php?from=sights"> <b>Bulk Copy:</b>      <select name='room' class="form-control">         <option>Select...</option>         <option value="Orhan">Orhan</option>         <option value="Deniz">Deniz</option>         <option value="Irini">Irini</option>         <option value="Katina">Katina</option>         <option value="Gulbin">Gulbin</option>         <option value="Mihalis">Mihalis</option>     </select>     <input class="btn btn-primary" type="submit" name="submit" value="Go"><br /><br /> </div> <table class="table table-bordered table-striped">     <th>Entry Name</th>     <th>Display Order</th>     <th>Copy to...</th>     <th>Status</th>     <th>Image</th>     <th>Edit</th>     <th>Delete</th>     <th>Duplicate</th>      <?php while($row = mysql_fetch_array($sel)) { ?>     <tr>         <td>             <input type="checkbox" name="id[]" value="<?php echo $row['id']; ?>">             </form>             <?php echo $row['title']; ?>         </td>         <td>             <form name="order" method="post" action="sightorder.php?id=<?php echo htmlspecialchars($row['id']); ?>">                 <div class="col-md-4">                     <input class="form-control" type="number" name="order" value="<?php echo htmlspecialchars($row['ordernum']); ?>">                 </div>                 <div class="col-sm-3">                     <input type="submit" name="submit" value="Set Order" class="btn btn-primary">                 </div>             </form>         </td>         <td>              <form name="copyto" method="post" action="copyto.php?from=sights&id=<?php echo htmlspecialchars($row['id']); ?>">                 <input type="checkbox" name="room[]" value="Orhan"> O -                 <input type="checkbox" name="room[]" value="Deniz"> D -                 <input type="checkbox" name="room[]" value="Irini"> I -                 <input type="checkbox" name="room[]" value="Katina"> K -                 <input type="checkbox" name="room[]" value="Gulbin"> G -                 <input type="checkbox" name="room[]" value="Mihalis"> M                  <input type="submit" name="submit" value="Copy" class="btn btn-primary">             </form>          </td>         <td>             <a href="sightstatus.php?id=<?php echo htmlspecialchars($row['id']); ?>&status=<?php echo $row['status']; ?>"><?php if($row['status'] == 1){ ?><i class="fa fa-check fa-lg"></i><?php }else{ ?><i class="fa fa-times fa-lg"></i><?php } ?></a>         </td>         <td>             <a href="sightimages.php?id=<?php echo $row['id']; ?>"><i class="fa fa-image fa-lg"></i></a>         </td>         <td>             <a href="editsight.php?id=<?php echo htmlspecialchars($row['id']); ?>"><i class="fa fa-edit fa-lg"></i></a>         </td>         <td>             <a onclick="return confirmDelete()" href="delsight.php?id=<?php echo htmlspecialchars($row['id']); ?>"><i class="fa fa-trash fa-lg"></i></a>         </td>         <td>             <a href="duplicatesight.php?id=<?php echo htmlspecialchars($row['id']); ?>"><i class="fa fa-copy fa-lg"></i></a>         </td>     </tr>     <?php } ?> </table> 

Any help would be greatly appreciated. Thanks.

8 Answers

Answers 1

You have a problem here

<?php     while($row = mysql_fetch_array($sel)){ ?>         <tr><td><input type="checkbox" name="id[]" value="<?php echo $row['id']; ?>"> <?php echo $row['title']; ?></td></form> 

There is no closing bracket for the while loop, and the form is closed after the first checkbox is added. So if that checkbox is not checked, then the input is not posted, thus the undefined index. Make sure you do not close the form until after all the rows have been added, like this

<?php     while($row = mysql_fetch_array($sel)){ ?>         <tr><td><input type="checkbox" name="id[]" value="<?php echo $row['id']; ?>"> <?php echo $row['title']; ?></td></tr> <?php } ?>   </table> </form> 

Answers 2

After reviewing the raw HTML of the complete page you provided, it is clear that the problem is you're trying to nest multiple forms which is invalid HTML. Please refer to this answer for more information. This answer does link to a workaround, but it's an ugly hack and should probably be avoided.

I believe the appropriate, valid HTML solution in your case is to use a single form. Currently you have multiple nested forms submitting to the following locations:

1. bulkcopy.php?from=sights
2. sightorder.php?id=1
3. copyto.php?from=sights&id=1
4. copyto.php?from=sights&id=46
5. etc...

What you can do is have a single form that determines which action to take based on which submit button was clicked. For example:

switch ($_POST['submit']) {     case 'Go':         // process bulkcopy         break;      case 'Set Order':         // process siteorder         break;      // etc... } 

Answers 3

if you the variable checkbox is a table $_POST['id'] the when you do this

foreach($_POST['id'] as $check) {   $id = $check;   ... } 

if you don't check the first input the first variable

$check = $_POST['id']['0'];  // is empty 

you can do another condition in the foreach

if(!empty($_POST['id'])) {   foreach($_POST['id'] as $k=>$v) {      if(!empty($v)){              $id = $v;         $sel = mysql_query("select * from $from where id = '$id' limit 1 ") or die(mysql_error());         while($row = mysql_fetch_array($sel)){            $preview = $row['preview'];            $text = $row['text'];            $title = $row['title'];            $images = $row['images'];        }         $ins = mysql_query("insert into $room (id, preview, text, title, images) values (' ', '$preview', '$text', '$title', '$images') ") or die(mysql_error());        }   }  header("Location:admin.php"); } 

Answers 4

You should use mysql_num_rows() to check if u actually have result before trying to access them and use them for insertion to database.The problem with your code is that the variables inside

 while($row = mysql_fetch_array($sel)){    ....    } 

are never defined in case the result set is empty.But although they are never defined you try to use them in an insert query later.So just check if you have results first:

 <?php session_start();  if($_SESSION['admin_logged_in'] != true){   header("Location:login.html");   exit(); }  include 'db.php';  $from = mysql_real_escape_string($_GET['from']); $room = mysql_real_escape_string($_POST['room']);  if(isset($_POST['id'])&&!empty($_POST['id'])) { foreach($_POST['id'] as $check) {   if(empty($check)) continue;   $id = $check;    $sel = mysql_query("select * from $from where id = '$id' limit 1 ") or die(mysql_error());  if(mysql_num_rows($sel)>0){   while($row = mysql_fetch_array($sel)){     $preview = $row['preview'];     $text = $row['text'];     $title = $row['title'];     $images = $row['images'];  }    $ins = mysql_query("insert into $room (id, preview, text, title, images) values (' ', '$preview', '$text', '$title', '$images') ") or die(mysql_error());   }  }  header("Location:admin.php"); }  ?> 

EDIT:

Also it seems there is a problem with your generated html code.You dont seem to close properly your tags.Try this:

         <?php          while($row = mysql_fetch_array($sel)){ ?>         <tr><td><input type="checkbox" name="id[]" value="<?php echo $row['id']; ?>"> <?php echo $row['title']; ?></td></tr><?php }?></table></form> 

But since i do know the value of $sel I cannot help you more if you do not post your generated html.

Answers 5

Your form does not POST an input with the name 'room'. Therefore, when you try and check it with this line $room = mysql_real_escape_string($_POST['room']);, there is no item in the $_POST array with the index 'room', hence the undefined index error.

To debug this kind of thing, it's helpful to analyse the request/response headers of your form submission. If you use Chrome, press Ctrl+Shift+I to bring up the developer console, select the Network tab, and when you submit your form, view the details of the entry that pops up. You will be able to see here the names and values of the things being sent and will give you an idea of where things are going wrong. Other browsers are available, and each have their own respectable versions of this.

Also, before accessing any variables you didn't define yourself, or can't rely on (such as form submissions), you should use isset() to make sure the variable exists before using it - that will stop the error you are getting and also allow you to catch where it's going wrong more easily:

if (!isset($_POST['room'])) {   print('Please select something'); } else {   $room = $_POST['room']; // technically not needed tho :p   //... } 

Answers 6

which is tha page you are callling, sightorder.php or copyto.php? by which submit button, set order or copy?

in both cases you are sending the id by GET not by POST.

which is the line which produces the error?

Luca

Answers 7

  <form bulkcopy>   ....    <table>   <?php while: ?>       // </form> it must be deleted       ...       <div sightorder>          ...          <input submit onclick="return send_sightorder(this);">       </div>        ...       <div copyto action="copyto.php?from=sights&id=<?php echo htmlspecialchars($row['id']); ?>">           ...           <input submit onclick="return send_copy(this);">       </div>    <?php end while ?>   </table>   </form> // bulkcopy's close tag    <script>      /// Function send_sightorder is same.      function send_copy(clicked_element) {          var form = clicked_element.parent;          var inputs = form.getElementsByTagName("input");          var data = inputs[0].name + "=" + inputs[0],checked + "&";          var URL = window.location.host + form.getAttribute("action");                for (i=1; i<inputs.length-1; ++i) {              data = "&" + inputs[i].name + "=" + inputs[i].checked + "&";          }           xmlhttp = new XMLHttpRequest();          xmlhttp.onreadystatechange = function() {             if (xmlhttp.readyState == 4 && xmlhttp.status == 200) {                // redirect to new page or something else ...                window.location = URL;             }          }          xmlhttp.open("POST", URL,true);          xmlhttp.send(data);           return false;      }   </script> 

Answers 8

the trick is to give your checkboxes fixed names with the value of the IDs

<input type="checkbox" name="id[<?php echo $row['id']; ?>]" value="1"> 

in your php code do like:

if (is_array($_POST['id'])) {      foreach ($_POST['id'] as $id => $val) {          // should always be 1 as most browsers don't send unchecked checkboxes...         // better to check it. also check for "on" because some browsers always send it as value for checked checkboxes         if (1 == $val || 'on' == $val) {               echo "Checkbox with ID ". $id . " was checked";         }     } } 

Read More