Add variable to nested list

Using R base, I would like to add a variable to a nested list where the variable changes for each nested list element. Below is an example. Thank you.

#CREATE EXAMPLE DATAFRAME DF <- expand.grid(NAME = c("FRANK", "TONY", "ED"), DATE = seq(as.Date("2014-01-01"), as.Date("2018-12-31"), by = "day"))  #CREATE NESTED LIST DF <- lapply(1:3, function(i) lapply(2014:2015, function(t) DF[with(DF, as.Date(paste(t, "01", "01", sep = "-")) <= DATE & DATE <= as.Date(paste(t + i, "12", "31", sep = "-"))), ]))  #PRINT NESTED LIST lapply(DF, lapply, function(x) rbind(head(x), tail(x)))  #I WOULD LIKE TO SIMPLIFY THIS PART DF[[1]][[1]] <- within(DF[[1]][[1]], GROUP <- 2014) DF[[1]][[2]] <- within(DF[[1]][[2]], GROUP <- 2015)  DF[[2]][[1]] <- within(DF[[2]][[1]], GROUP <- 2014) DF[[2]][[2]] <- within(DF[[2]][[2]], GROUP <- 2015)  DF[[3]][[1]] <- within(DF[[3]][[1]], GROUP <- 2014) DF[[3]][[2]] <- within(DF[[3]][[2]], GROUP <- 2015)  #PRINT MODIFIED NESTED LIST lapply(DF, lapply, function(x) rbind(head(x), tail(x)))  #I AM SURPRISED THE FOLLOWING DOES NOT WORK DF <- expand.grid(NAME = c("FRANK", "TONY", "ED"), DATE = seq(as.Date("2014-01-01"), as.Date("2018-12-31"), by = "day")) DF <- lapply(1:3, function(i) lapply(2014:2015, function(t) DF[with(DF, as.Date(paste(t, "01", "01", sep = "-")) <= DATE & DATE <= as.Date(paste(t + i, "12", "31", sep = "-"))),])) DF <- lapply(DF, function(x) lapply(2014:2015, function(t) within(x, GROUP <- t))) lapply(DF, lapply, function(x) rbind(head(x), tail(x))) 

6 Answers

Answers 1

This should do it

final_list<-list() for(i in seq(1, length(DF))){    new_list<-list()    for(j in seq(1,length(DF[[i]]))){      new_list[[j]]<-list(DF[[i]][[j]],GROUP=j)    }   final_list[[i]]<-new_list } 

Answers 2

#CREATE EXAMPLE DATAFRAME DF <- expand.grid(NAME = c("FRANK", "TONY", "ED"), DATE = seq(as.Date("2014-01-01"), as.Date("2018-12-31"), by = "day"))  #CREATE NESTED LIST DF <- lapply(1:3, function(i) lapply(2014:2015, function(t) DF[with(DF, as.Date(paste(t, "01", "01", sep = "-")) <= DATE & DATE <= as.Date(paste(t + i, "12", "31", sep = "-"))), ]))  #PRINT NESTED LIST lapply(DF, lapply, function(x) rbind(head(x), tail(x)))  #I WOULD LIKE TO SIMPLIFY THIS PART DF[[1]][[1]] <- within(DF[[1]][[1]], GROUP <- 2014) DF[[1]][[2]] <- within(DF[[1]][[2]], GROUP <- 2015)  DF[[2]][[1]] <- within(DF[[2]][[1]], GROUP <- 2014) DF[[2]][[2]] <- within(DF[[2]][[2]], GROUP <- 2015)  DF[[3]][[1]] <- within(DF[[3]][[1]], GROUP <- 2014) DF[[3]][[2]] <- within(DF[[3]][[2]], GROUP <- 2015)  #PRINT MODIFIED NESTED LIST DF1 <- lapply(DF, lapply, function(x) rbind(head(x), tail(x))) DF1  #I AM SURPRISED THE FOLLOWING DOES NOT WORK DF <- expand.grid(NAME = c("FRANK", "TONY", "ED"), DATE = seq(as.Date("2014-01-01"), as.Date("2018-12-31"), by = "day")) DF <- lapply(1:3, function(i) lapply(2014:2015, function(t) DF[with(DF, as.Date(paste(t, "01", "01", sep = "-")) <= DATE & DATE <= as.Date(paste(t + i, "12", "31", sep = "-"))),]))  GROUPS <- c(2014:2015)      DF <- lapply(DF, function(xs) lapply(1:2, function(t) within(xs[[t]], GROUP <- GROUPS[t])))  DF2 <- lapply(DF, lapply, function(x) rbind(head(x), tail(x))) DF2  all.equal(DF1, DF2) 

Answers 3

How about this? Had to use package dplyr since it makes adding a new variable to the dataframes easier with mutate_.

library(dplyr) # if not installed, install with install.packages("dplyr") DF <- expand.grid(NAME = c("FRANK", "TONY", "ED"),  DATE = seq(as.Date("2014-01-01"), as.Date("2018-12-31"), by = "day"))  #CREATE NESTED LIST DF <- lapply(1:3, function(i) lapply(2014:2015, function(t) DF[with(DF,  as.Date(paste(t, "01", "01", sep = "-")) <= DATE & DATE <= as.Date(paste(t +  i, "12", "31", sep = "-"))), ]))  # loop over the first list with lapply and then loop over the nested lists  # and the desired GROUP values with mapply DF <- lapply(DF, function(x) mapply(FUN = function(df,number){mutate_(df,  "GROUP" = number)},x, 2014:2015, SIMPLIFY = F))  #PRINT NESTED LIST lapply(DF, lapply, function(x) rbind(head(x), tail(x))) 

As to why your way was not working: Think about what function(t) within(xs[[t]], GROUP <- GROUPS[t]) does. It does not return a dataframe.

Answers 4

This can be also achieved using Map in order to cbind the dates. i.e.,

lapply(DF, function(i) Map(cbind, i, c(2014, 2015)))  #or to set the name of that column to 'id',  lapply(DF, function(i) Map(function(x, y) cbind(x, id = y), i, c(2014, 2015))) 

However, If you don't mind flattening that list and keeping an extra id variable, you can use tidyverse to flatten and create the year id by graping the year from the first element of DATE from each data frame, i.e.

library(tidyverse)  new_df <- DF %>%    flatten_df(.id = 'list_id') %>%    group_by(list_id) %>%    mutate(id = sub('-.*', '', DATE[1]))  #which will give,  # A tibble: 19,725 x 4 # Groups:   list_id [6] #   list_id   NAME       DATE    id #     <chr> <fctr>     <date> <chr> # 1       1  FRANK 2014-01-01  2014 # 2       1   TONY 2014-01-01  2014 # 3       1     ED 2014-01-01  2014 # 4       1  FRANK 2014-01-02  2014 # 5       1   TONY 2014-01-02  2014 # 6       1     ED 2014-01-02  2014 # 7       1  FRANK 2014-01-03  2014 # 8       1   TONY 2014-01-03  2014 # 9       1     ED 2014-01-03  2014 #10       1  FRANK 2014-01-04  2014 # ... with 19,715 more rows 

Answers 5

I think the problem may the second lapply:

DF <- lapply(DF, function(x) lapply(2014:2015, function(t) within(x, GROUP <- t))) 

The lapply's don't appear to be extracting the desired component from the large list object. The first lapply iterates over the top level of the list, extracting a two-element list object, x, each time. The second lapply then iterates over a vector, providing a scalar vector, t, each time. So, the next part gets a two-element list (x) each time instead of the desired (unnamed) data frame.

Solution

If the object has already been created, you can iterate over the elements directly instead of indexing the list elements.

#CREATE NESTED LIST DF <- lapply(1:3, function(i) lapply(2014:2015, function(t) DF[with(DF, as.Date(paste(t, "01", "01", sep = "-")) <= DATE & DATE <= as.Date(paste(t + i, "12", "31", sep = "-"))), ]))  edit_level2 <- function(df) {     # figure out what the value of t should be based on the data.     t <- as.integer(format(min(df$DATE), "%Y"))     df$GROUP <- t     return(df) }  # iterate over the list object contents at *both* levels DF <- lapply(DF, function(level1) lapply(level1, function(level2) edit_level2(level2))) 

Note: this is similar to the solution presented by @Consistency in the comments--extracting the data frame is the issue.

Alternative

If you can you alter the code that is producing the list object, I suggest assigning the variable while creating the list object instead of modifying it afterwards (my original suggestion before edits).

#CREATE EXAMPLE DATAFRAME DF <- expand.grid(NAME = c("FRANK", "TONY", "ED"),                    DATE = seq(as.Date("2014-01-01"), as.Date("2018-12-31"), by = "day"))  #CREATE NESTED LIST DF <- lapply(1:3, function(i) {      lapply(2014:2015, function(t) {          first <- as.Date(paste(t,     "01", "01", sep = "-"))          last  <- as.Date(paste(t + i, "12", "31", sep = "-"))           # create a local data frame         df <- DF[first <= DF$DATE & DF$DATE <= last, ]          # modify the local data frame         df$GROUP <- t          # return the modified data frame         df      })  }) 

Answers 6

OK, I think I've got it. At least all.equal returns TRUE, though identical does not. Here it goes. Note that I've changed your df names a bit.

#CREATE EXAMPLE DATAFRAME DF <- expand.grid(NAME = c("FRANK", "TONY", "ED"), DATE =    seq(as.Date("2014-01-01"), as.Date("2018-12-31"), by = "day"))  #CREATE NESTED LIST DF <- lapply(1:3, function(i) lapply(2014:2015, function(t) DF[with(DF, as.Date(paste(t, "01", "01", sep = "-")) <= DATE & DATE <= as.Date(paste(t + i, "12", "31", sep = "-"))), ]))  #PRINT NESTED LIST lapply(DF, lapply, function(x) rbind(head(x), tail(x)))  DF2 <- DF #I WOULD LIKE TO SIMPLIFY THIS PART DF2[[1]][[1]] <- within(DF2[[1]][[1]], GROUP <- 2014) DF2[[1]][[2]] <- within(DF2[[1]][[2]], GROUP <- 2015)  DF2[[2]][[1]] <- within(DF2[[2]][[1]], GROUP <- 2014) DF2[[2]][[2]] <- within(DF2[[2]][[2]], GROUP <- 2015)  DF2[[3]][[1]] <- within(DF2[[3]][[1]], GROUP <- 2014) DF2[[3]][[2]] <- within(DF2[[3]][[2]], GROUP <- 2015)  #PRINT MODIFIED NESTED LIST lapply(DF2, lapply, function(x) rbind(head(x), tail(x)))  ### New code DF3 <- DF DF3 <- lapply(DF3, function(x) {         lapply(2014:2015, function(t){             within(x[[t - 2013]], GROUP <- t)         })     }) identical(DF2, DF3) all.equal(DF2, DF3) 

Read More

Elastic search: find word parts in nested object

I'm not able to find parts of a word in a nested object. Only the full word is found. My analyzer configuration is as follows:

{   "settings": {     "number_of_shards": 1,     "analysis": {       "filter": {         "word_part_filter": {           "type": "ngram",           "min_gram": 3,           "max_gram": 15         },         "word_part_front_filter": {           "type": "edgeNGram",           "min_gram": 2,           "max_gram": 15         },         "codeid_filter": {           "type": "pattern_replace",           "pattern": "[-/.:]",           "replacement": "",           "preserve_original": true         }       },       "char_filter": {         "umlaut_char_filter": {           "type": "mapping",           "mappings": [             "ö=>oe",             "ä=>ae",             "ü=>ue",             "�=>ss",             "�=>Oe",             "�=>Ae",             "�=>Ue"           ]         }       },       "analyzer": {         "description_analyser_query": {           "type": "custom",           "char_filter": [             "html_strip"           ],           "tokenizer": "standard",           "filter": [             "lowercase",             "stop",             "asciifolding"           ]         },         "description_analyser_idx": {           "type": "custom",           "char_filter": [             "html_strip"           ],           "tokenizer": "standard",           "filter": [             "lowercase",             "stop",             "asciifolding",             "word_part_filter"           ]         },         "name_analyser_query": {           "type": "custom",           "char_filter": [             "umlaut_char_filter"           ],           "tokenizer": "standard",           "filter": [             "lowercase",             "asciifolding"           ]         },         "name_analyser_idx": {           "type": "custom",           "char_filter": [             "umlaut_char_filter"           ],           "tokenizer": "standard",           "filter": [             "lowercase",             "asciifolding",             "word_part_filter"           ]         },         "codeid_analyser_query": {           "type": "custom",           "tokenizer": "keyword",           "filter": [             "lowercase",             "codeid_filter"           ]         },         "codeid_analyser_idx_front": {           "type": "custom",           "tokenizer": "keyword",           "filter": [             "lowercase",             "codeid_filter",             "word_part_front_filter"           ]         },         "codeid_analyser_idx_any": {           "type": "custom",           "tokenizer": "keyword",           "filter": [             "lowercase",             "codeid_filter",             "word_part_filter"           ]         }       }     }   } } 

This is the nested object mapping (extracted):

{   "properties": {         "aid": {       "type": "nested",       "properties": {         "tpid": {           "type": "string",           "analyzer": "codeid_analyser_idx_any"         },         "aid": {           "type": "string",           "analyzer": "codeid_analyser_idx_any"         }       }         }   } } 

I'm searching with this query (extract). Only the "nested" part is essential here:

{   "query": {     "bool": {       "must": [         {           "bool": {             "should": [               {                 "nested": {                   "path": "aid",                   "query": {                     "bool": {                       "must": {                         "match": {                           "aid.aid": {                             "query": "1200",                             "analyzer": "codeid_analyser_query"                           }                         }                       },                       "filter": {                         "or": [                           {                             "match": {                               "aid.tpid": "buyer_specific"                             }                           },                           {                             "match": {                               "aid.tpid": "mytpid"                             }                           }                         ]                       }                     }                   }                 }               }             ],             "minimum_should_match": 1           }         }       ]     }   } } 

there is an element with aid=120000008

When using the analyzers in the fields, it finds nothing. When using no analyzers at all in the nested object mapping and query, only full words (like "120000008") will be found, but not "1200". Any ideas?

1 Answers

Answers 1

Actually, using ElasticSearch 5.2, using an index called test, and applying the mapping on a type called "product" (rewriting only the filter part so it is compliant to the evolution of the querying language), I obtain the correct result. The query:

GET test/_search {   "query": {     "bool": {       "must": [         {           "bool": {             "should": [               {                 "nested": {                   "path": "aid",                   "query": {                     "bool": {                       "must": {                         "match": {                           "aid.aid": {                             "query": "1200",                             "analyzer": "codeid_analyser_query"                           }                         }                       },                       "filter": {                         "terms": {                           "aid.tpid": [                             "mytpid",                             "buyer_specific"                           ]                         }                       }                     }                   }                 }               }             ],             "minimum_should_match": 1           }         }       ]     }   } } 

The index:

GET test/_search  {   "took": 8,   "timed_out": false,   "_shards": {     "total": 5,     "successful": 5,     "failed": 0   },   "hits": {     "total": 1,     "max_score": 1,     "hits": [       {         "_index": "test",         "_type": "product",         "_id": "AVrJ1CSd-NyeQ4r64kP6",         "_score": 1,         "_source": {           "aid": {             "aid": "120000008",             "tpid": "mytpid"           }         }       }     ]   } } 

The analyzer (I removed the umlaut filter as it was unreadable on my computer, and it does not change the result as in the tests it is not used):

PUT test {   "settings": {      "analysis": {       "filter": {         "word_part_filter": {           "type": "ngram",           "min_gram": 3,           "max_gram": 15         },         "word_part_front_filter": {           "type": "edgeNGram",           "min_gram": 2,           "max_gram": 15         },         "codeid_filter": {           "type": "pattern_replace",           "pattern": "[-/.:]",           "replacement": "",           "preserve_original": true         }       },        "analyzer": {         "description_analyser_query": {           "type": "custom",           "char_filter": [             "html_strip"           ],           "tokenizer": "standard",           "filter": [             "lowercase",             "stop",             "asciifolding"           ]         },         "description_analyser_idx": {           "type": "custom",           "char_filter": [             "html_strip"           ],           "tokenizer": "standard",           "filter": [             "lowercase",             "stop",             "asciifolding",             "word_part_filter"           ]         },          "codeid_analyser_query": {           "type": "custom",           "tokenizer": "keyword",           "filter": [             "lowercase",             "codeid_filter"           ]         },         "codeid_analyser_idx_front": {           "type": "custom",           "tokenizer": "keyword",           "filter": [             "lowercase",             "codeid_filter",             "word_part_front_filter"           ]         },         "codeid_analyser_idx_any": {           "type": "custom",           "tokenizer": "keyword",           "filter": [             "lowercase",             "codeid_filter",             "word_part_filter"           ]         }       }      }   } } 

The mapping on product:

PUT test/_mapping/product {    "properties": {         "aid": {       "type": "nested",       "properties": {         "tpid": {           "type": "string",           "analyzer": "codeid_analyser_idx_any"         },         "aid": {           "type": "string",           "analyzer": "codeid_analyser_idx_any"         }       }         }   } } 

Read More

gradle to bundle nested jar dependencies into module output jar

How to make gradle to included some dependencies into module resulted jar as jar? i.e. to make jar with nested jar's e.g. in lib folder ?

This is not Android project, and this should be done for many modules in multi-module project, so fatJar, uberJar, shadowJar alike solutions seem not to fit.

1 Answers

Answers 1

You just need to add an additional from directive to include dependencies in your jar:

task jarJar(type: Jar) {     baseName = project.name + '-jarjar'     from { configurations.compile }     with jar } 

Read More

How to handle different reference directions in database and ZF2 application?

Zend\Form\Fieldsets and Zend\Form\Collectionss can be nested and provide a very comfortable way to map complex object structures to them, in order to get a comlete object (ready to be saved) from the form input more or less automatically. The Form Collections tutorial provides a very good example.

The case I'm currently having is a bit more complex, since it contains a reference inversion. That means:

I have two entities -- MyA and MyB and while in the database the relationship between them is implemented as FOREIGN KEY from myb.mya_id to mya.id, the application is using an inverted referencing:

MyA has MyB 

Or with some code:

namespace My\DataObject;  class MyA {     /**      * @var integer      */     private $id;     /*      * @var text      */     private $foo;     /**      * @var MyB      */     private $myB; }  namespace My\DataObject;  class MyB {     /**      * @var integer      */     private $id;     /*      * @var text      */     private $bar;     /*     Actually it's even bidirectional, but it's not crucial for this issue.     For this problem it's not important,     wheter the class MyB has a property of type MyA.     We get the issue already,     when we define a property of type MyB in the class MyA.     Since the direction of the reference MyA.myB->MyB differes     from the direction of the reference my_b.my_a.id->my_a.id.     */      /**      * @var MyA      */     // private $myA; } 

My Mapper objects get DataObjects passed as argument: MyAMapper#save(MyA $object) and MyBMapper#save(MyB $object).

namespace My\Mapper; use ... class MyAMapper {     ...     public fuction save(MyA $object)     {         // save the plain MyA propertis a new entry in the my_a table         ...         $myMapperB->save($myA->getMyB());     } }  namespace My\Mapper; use ... class MyBMapper {     ...     public fuction save(MyB $object)     {         // save the plain MyB propertis a new entry in the my_b table         ...     } } 

That means, the MyAMapper#save(...) has evrything needed to save the MyA object to the my_a table. But in the MyBMapper the data for my_b.my_a_id will be missing.

And I also cannot create a fieldset MyAFieldset with a nested fieldset MyBFieldset and then nest the fieldset MyBFieldset into MyAFieldset in order to fill MyA#MyB#MyA (in order to pass the data for my_b.my_a_id to MyBMapper#save(...)):

class MyAFieldset {     $this->add([         'name' => 'my_b',         'type' => 'My\Form\Fieldset\MyBFieldset',         'options' => []     ]); }  class MyBFieldset {     $this->add([         'name' => 'my_a',         'type' => 'My\Form\Fieldset\MyAFieldset',         'options' => []     ]); } 

This would cause a recursive dependency and cannot work.

How to handle a case, when the reference direction on the application level differs from it's direction in the database? How to create though a fieldsets structure, that provides a complete ("ready to be saved") object?

---

Workaround 1

When the form is processed, a further MyA object can be created and added to the MyB object got from the form:

class MyConrtoller {     ...     public function myAction() {         $this->myForm->bind($this->myA);         $request = $this->getRequest();         $this->myForm->setData($request->getPost());         // here the hack #start#         $this->myB->setMyA($this->myA);         // here the hack #stop#         $this->myAService->saveMyA($this->myA);     } } 

Well, maybe not in the controller, the mapper might be a better place for that:

class MyAMapper {     ...     public function save(MyA $myA)     {         $data = [];         $data['foo'] = [$myA->getFoo()];         // common saving stuff #start#         $action = new Insert('my_a');         $action->values($data);         $sql = new Sql($this->dbAdapter);         $statement = $sql->prepareStatementForSqlObject($action);         $result = $statement->execute();         $newId = $result->getGeneratedValue()         // common saving stuff #stop#         ...         // hack #start#         if(! $myA->getB->getA()) {             $myA->getB->setA(new MyA());             $myA->getB->getA()->setId($newId);         }         // hack #stop#         // and only after all that we can save the MyB         $myB = $this->myBMapper->save($myB);         $myA->setMyB($myB);         ...     } } 

But anyway it's just a hack.

Workaround 2

The MyB class gets a property $myAId. But it's also not a clean way.

Workaround 3

The MyBFieldset gets a MyAFieldsetFake as sub-fieldset. This fieldset class is then just a "shallow" copy of the MyAFieldset, that contains only the ID for the MyA data object:

class MyAFieldset {     ...     public function init()     {         $this->add([             'type' => 'text',             'name' => 'id',             'options' => [...],         ]);         $this->add([             'type' => 'text',             'name' => 'foo',             'options' => [...],         ]);     } } class MyAFieldset {     ...     public function init()     {         $this->add([             'type' => 'text',             'name' => 'id',             'options' => [...],         ]);         $this->add([             'type' => 'text',             'name' => 'bar',             'options' => [...],         ]);         $this->add([             'type' => 'text',             'name' => 'foo',             'type' => 'My\Form\Fieldset\MyAFakeFieldset',             'options' => [...],         ]);     } } class MyAFieldset {     ...     public function init()     {         $this->add([             'type' => 'text',             'name' => 'id',             'options' => [...],         ]);     } } 

But fake objects are a bit dirty as well.

1 Answers

Answers 1

How about creating a new table to handle the mappings on their own. Then you can isolate that complexity away from the objects that take advantage of them.

So, you could have a new object AtoBMappings

namespace My\DataObject;  class MyA {     /**      * @var integer      */     private $id;     /*      * @var text      */     private $foo;     /**      * @var MyAtoB      */     private $myAtoB; }  namespace My\DataObject;  class MyB {     /**      * @var integer      */     private $id;      /**      * @var AtoBMapperID      */     private $myAtoB; }  class MyAtoBMapper {    /**     * @var myB     */    private $myB    /**     * @var myA    **    private $myA } 

Then, instead of hacking your Mapper method, you can simply make an assignment in MyA to MyB creation.

class MyAMapper {     ...     public function save(MyA $myA)     {          $myAtoB = new MyAtoBMapper();         //.... instert new myAtoB into DB           $data = [];         $data['foo'] = [$myA->getFoo()];         $data['myAtoB'] = $myAtoB->getId();         // common saving stuff #start#         $action = new Insert('my_a');         $action->values($data);         $sql = new Sql($this->dbAdapter);         $statement = $sql->prepareStatementForSqlObject($action);         $result = $statement->execute();         $newId = $result->getGeneratedValue();         $myA->setMyAtoB($newAtoB);         $myAtoBMapper->myA = $newId;         // common saving stuff #stop#         // and only after all that we can save the MyB         $myB = $this->myBMapper->save($myB);         $myB->setMyAtoB($newAtoB);         $myAtoBMapper->myB = $myB;         ...     } } 

Do you think this would work, or do you think this is too much of a hack?

Read More