Elasticsearch sort: by one from each group and repeat

I need to get items with max value from each name and repeat until end.

I'll explain it on simple example. I have such items:

Name| Value ----------- AAA | 12 AAA | 35 AAA | 5 BBB | 1 BBB | 10 BBB | 5 

Expected result after sort:

Name| Value ----------- AAA | 35 BBB | 10 AAA | 12 BBB | 5 AAA | 5 BBB | 1 

I know how to do it in Postgres (window functions: rank() over()), but is it possible in Elastic?

3 Answers

Answers 1

You have to do something like Group by max

Here is Example

GET /yourindex/_search { "size": 0   "aggs": {     "yourGroup": {       "terms": {         "field": "Name",         "size": 10       },       "aggs": {         "theMax": {           "max": {             "field": "Value"           }         }       }     }   } } 

Reference:- this

Answers 2

Aggregating here my comments.

Showing >1 million records is a bad idea no matter how those documents are sorted, when it comes to Elasticsearch. My questions were asked to see how appropriate is to create a second ES index with the results of probably 1 query + post processing and holding something like "first 1000 records" (meaning a human reasonable list of documents) and to update that list periodically (every 10 seconds or so). You could use Watcher to build this index and keep it updated. 1 million records, as I said, is both impractical (who would look at 1mil docs) and not performant from ES point of view.

Basically, keep a separate index which should include only first 1000 documents that are sorted according to your requirements. And this index is updated regularly, not your main one with 1mil documents. Regarding pagination and 1 mil. documents... how many pages do you believe your users will go through?! 10, 15, 20? Not even google.com is giving you everything. Only few tens of pages, even though there can be hundreds of millions of matches. Keep in mind that Elasticsearch is a search engine, not a database. The aim is to give you the best matching docs, not all of them.

The query from Watcher will run over all the documents in your main index. It will aggregate the documents according to your requirements (I think a terms aggregation on Name, ordered by Value), you can add a post-processing step to create the order you need and then index that into a second index. Next time the watch will trigger, it will delete the old index, perform the same query again and index the new results in the (now empty) index.

Answers 3

Elasticsearch supports sorting by array or multi-valued fields. The mode option controls what array value is picked for sorting the document it belongs to. The mode option can have the following values:

min -- Pick the lowest value.

max -- Pick the highest value.

sum -- Use the sum of all values as sort value. Only applicable for number based array fields.

avg -- Use the average of all values as sort value. Only applicable for number based array fields.

median -- Use the median of all values as sort value. Only applicable for number based array fields.

Sort Order: The order option can have the following values:

asc -- Sort in ascending order

desc -- Sort in descending order

Below is a example in which field price has multiple prices per document. In this case the result hits will be sorted by price descending based on the average price per document.

PUT /my_index/_doc/1?refresh {    "product": "chocolate",    "price": [20, 4] }  POST /_search {    "query" : {       "term" : { "product" : "chocolate" }    },    "sort" : [       {"price" : {"order" : "desc", "mode" : "avg"}}    ] } 

Nested sorting example In the below example offer is a field of type nested. The nested path needs to be specified; otherwise, Elasticsearch doesn’t know on what nested level sort values need to be captured.

POST /_search {    "query" : {       "term" : { "product" : "chocolate" }    },    "sort" : [        {           "offer.price" : {              "mode" :  "avg",              "order" : "asc",              "nested": {                 "path": "offer",                 "filter": {                    "term" : { "offer.color" : "blue" }                 }              }           }        }     ] } 

Please refer this link Elastic search sort for detailed explanation and much more examples.

Read More

Sort string with integers and words without any change in their positions

Say I have a string a.

a = "12 I have car 8 200 a" 

I need to sort this string in such a way that the output should be

8 a car have 12 200 I 

ie, Sort the string in such a way that all words are in alphabetical order and all integers are in numerical order. Furthermore, if the nth element in the string is an integer it must remain an integer, and if it is a word it must remain a word.

This is what I tried.

a = "12 I have car 8 200 a"   def is_digit(element_):     """     Function to check the item is a number. We can make using of default isdigit function     but it will not work with negative numbers.     :param element_:     :return: is_digit_     """     try:         int(element_)         is_digit_ = True     except ValueError:         is_digit_ = False      return is_digit_    space_separated = a.split()  integers = [int(i) for i in space_separated if is_digit(i)] strings = [i for i in space_separated if i.isalpha()]  # sort list in place integers.sort() strings.sort(key=str.lower)  # This conversion to iter is to make use of next method. int_iter = iter(integers) st_iter = iter(strings)  final = [next(int_iter) if is_digit(element) else next(st_iter) if element.isalpha() else element for element in          space_separated]  print " ".join(map(str, final)) # 8 a car have 12 200 I 

I am getting the right output. But I am using two separate sorting function for sorting integers and the words(which I think is expensive).

Is it possible to do the entire sorting using a single sort function?.

6 Answers

Answers 1

numpy allows to write it more concisely, though doesn't eliminate the need for two separate sorts:

$ python3 Python 3.5.2 (default, Nov 23 2017, 16:37:01)  [GCC 5.4.0 20160609] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> from numpy.core.defchararray import isdecimal, lower >>>  >>> s = "12 I have car 8 200 a" >>>  >>> a = np.array(s.split()) >>>  >>> integer_mask = isdecimal(a) >>> string_mask = ~integer_mask >>> strings = a[string_mask] >>>  >>> a[integer_mask] = np.sort(np.int_(a[integer_mask])) >>> a[string_mask]  = strings[np.argsort(lower(strings))] >>>  >>> ' '.join(a) '8 a car have 12 200 I' 

Answers 2

Is it possible to do the entire sorting using a single sort function?.

No, not really.

Why not? It turns out the answer is already in your code.

integers.sort() strings.sort(key=str.lower) 

Notice how you need to sort by two different functions here. The first is an integer sort, the second is a lowercase string sort. We could try something like this:

def get_sort_order(element):     try:         value = int(element)     except ValueError:         value = element.lower()     return value  a.sort(key=get_sort_order) 

But that doesn't work either; it just gives us the result

['8', '12', '200', 'a', 'car', 'have', 'I'] 

You could probably force this into a solution, but it isn't going to be pretty.

However, there is another point I'd like to address:

But I am using two separate sorting function for sorting integers and the words (which I think is expensive).

Sorting two distinct lists is basically always going to be faster anyway. To find out why, just look at the time complexity of the two tasks:

Assuming a list of length 1000, exactly half integer and half strings, and a sorting algorithm of O(nlog(n)):

One single sort: 1000 * log(1000) = 3000

Two separate sorts: 2 * (500 * log(500) = ~2699

So sorting the list in a single run is both more difficult to implement and slower!

Answers 3

It is possible in one sort, by applying a custom function within the 'sorted' method as a User described above. I have tried a simplified version for the same. The default 'sorted' method does the wonder with a little tweak. Hope it resolves your query.

import re  input = "12 I have car 8 200 a" splitted = input.split() s_lst=sorted(splitted, key=lambda a:int(a) if a.isdigit() else a.lower())  count_nos = re.findall(r'\d+',' '.join(s_lst)) str_index = len(count_nos) no_index = 0 result=[] for i in range(0,len(splitted)):     if splitted[i].isdigit():         result.append(s_lst[no_index])         no_index+=1     else:         result.append(s_lst[str_index])         str_index+=1 print ' '.join(result) 

Answers 4

You could do in one sort provided you write a custom function for comparision. The idea is to sort the words in ascending order and integer in descending order in the same list . Incase of word and integer is compared then treat the word as smaller compared to word.

And then for printing the final result increment the index for word if a word is found , decrement the index for integer if digit is found.

The below code works in python2:

a = "12 I have car 8 200 a"  def custom_compare(x,y):     if x.isdigit() and y.isdigit():         return int(y) - int(x) #do a descending order     if x.isdigit() and y.isdigit() == False:         return 1     if x.isdigit() == False and y.isdigit():         return -1     if x.isdigit() == False and y.isdigit() == False:         # do ascending order         if x.lower() == y.lower():             return 0         elif x.lower() < y.lower():             return -1         else:             return 1  original_list = a.split(" ") sorted_list = sorted(original_list, cmp=custom_compare)  result = [] integer_index = -1 string_index = 0 for word in original_list:     if word.isdigit():         result.append(sorted_list[integer_index])         integer_index = integer_index - 1     else:         result.append(sorted_list[string_index])         string_index = string_index + 1  result ['8', 'a', 'car', 'have', '12', '200', 'I'] 

Python 3: import functools

a = "12 I have car 8 200 a"  def custom_compare(x,y):     if x.isdigit() and y.isdigit():         return int(y) - int(x) #do a descending order     if x.isdigit() and y.isdigit() == False:         return 1     if x.isdigit() == False and y.isdigit():         return -1     if x.isdigit() == False and y.isdigit() == False:         # do ascending order         if x.lower() == y.lower():             return 0         elif x.lower() < y.lower():             return -1         else:             return 1  original_list = a.split(" ") sorted_list = sorted(original_list, key=functools.cmp_to_key(custom_compare))  result = [] integer_index = -1 string_index = 0 for word in original_list:     if word.isdigit():         result.append(sorted_list[integer_index])         integer_index = integer_index - 1     else:         result.append(sorted_list[string_index])         string_index = string_index + 1  result 

PS:The word comparison could be efficiently written. I am from C background and I am not sure of the pythonic way of comparison.

Answers 5

s = "2 is a A -3 car 11 I 0 a"  def magick(s):   s = s.split()    def reverse(tuples):     return [(a, b) for (b, a) in tuples]    def do_sort(tuples):     firsts  = [a for a, _ in tuples]     seconds = [a for _, a in tuples]     return list(zip(sorted(firsts), seconds))    def str_is_int(x):     try:       int(x)       return True     except:       return False    indexed = list(enumerate(s))    ints = do_sort([(int(x), ix) for (ix, x) in indexed if     str_is_int(x)])   strs = do_sort([(    x , ix) for (ix, x) in indexed if not str_is_int(x)])    return ' '.join([str(b) for _, b in sorted(reverse(ints+strs))])  print(magick(s)) 

Answers 6

This solution utilizes a single custom sorting algorithm, after grouping the original input into integers and strings:

def gt(a, b):   return a > b if isinstance(a, int) and isinstance(b, int) else a[0].lower() > b[0].lower()  def type_sort(d):    '''similar to bubble sort, but does not swap elements of different types.        For instance, type_sort([5, 3, 'b', 'a']) => [3, 5, 'a', 'b']    '''    for _ in d:      for i in range(len(d)-1):        _c = d[i]        _t = d[i+1]        if isinstance(_c, type(_t)):          if gt(_c, _t):            d[i+1] = _c            d[i] = _t    return d  def get_type(x):   return int(x) if x.isdigit() else x  def sort_in_place(s:str):   _s = list(map(get_type, s.split()))   new_s = type_sort([i for i in _s if isinstance(i, int)]+[i for i in _s if isinstance(i, str)])   ints = iter(i for i in new_s if isinstance(i, int))   strings = iter(i for i in new_s if isinstance(i, str))   return ' '.join(map(str, [next(ints) if isinstance(i, int) else next(strings) for i in _s]))  print(sort_in_place(a)) 

Output:

'8 a car have 12 200 I' 

Read More

Nested lambda statements when sorting lists

I wish to sort the below list first by the number, then by the text.

lst = ['b-3', 'a-2', 'c-4', 'd-2']  # result: # ['a-2', 'd-2', 'b-3', 'c-4'] 

Attempt 1

res = sorted(lst, key=lambda x: (int(x.split('-')[1]), x.split('-')[0])) 

I was not happy with this since it required splitting a string twice, to extract the relevant components.

Attempt 2

I came up with the below solution. But I am hoping there is a more succinct solution via Pythonic lambda statements.

def sorter_func(x):     text, num = x.split('-')     return int(num), text  res = sorted(lst, key=sorter_func) 

I looked at Understanding nested lambda function behaviour in python but couldn't adapt this solution directly. Is there a more succinct way to rewrite the above code?

7 Answers

Answers 1

There are 2 points to note:

• One-line answers are not necessarily better. Using a named function is likely to make your code easier to read.
• You are likely not looking for a nested lambda statement, as function composition is not part of the standard library (see Note #1). What you can do easily is have one lambda function return the result of another lambda function.

Therefore, the correct answer can found in Lambda inside lambda.

For your specific problem, you can use:

res = sorted(lst, key=lambda x: (lambda y: (int(y[1]), y[0]))(x.split('-'))) 

Remember that lambda is just a function. You can call it immediately after defining it, even on the same line.

Note #1: The 3rd party toolz library does allow composition:

from toolz import compose  res = sorted(lst, key=compose(lambda x: (int(x[1]), x[0]), lambda x: x.split('-'))) 

Note #2: As @chepner points out, the deficiency of this solution (repeated function calls) is one of the reasons why PEP-572 is being considered.

Answers 2

We can wrap the list returned by split('-') under another list and then we can use a loop to handle it:

# Using list-comprehension >>> sorted(lst, key=lambda x: [(int(num), text) for text, num in [x.split('-')]]) ['a-2', 'd-2', 'b-3', 'c-4'] # Using next() >>> sorted(lst, key=lambda x: next((int(num), text) for text, num in [x.split('-')])) ['a-2', 'd-2', 'b-3', 'c-4'] 

Answers 3

lst = ['b-3', 'a-2', 'c-4', 'd-2'] res = sorted(lst, key=lambda x: tuple(f(a) for f, a in zip((int, str), reversed(x.split('-'))))) print(res)  ['a-2', 'd-2', 'b-3', 'c-4'] 

Answers 4

lst = ['b-3', 'a-2', 'c-4', 'd-2'] def xform(l):     return list(map(lambda x: x[1] + '-' + x[0], list(map(lambda x: x.split('-'), lst)))) lst = sorted(xform(lst)) print(xform(lst)) 

See it here I think @jpp has a better solution, but a fun little brainteaser :-)

Answers 5

you could convert to integer only if the index of the item is 0 (when reversing the splitted list). The only object (besides the result of split) which is created is the 2-element list used for comparison. The rest are just iterators.

sorted(lst,key = lambda s : [x if i else int(x) for i,x in enumerate(reversed(s.split("-")))]) 

As an aside, the - token isn't particularly great when numbers are involved, because it complicates the use of negative numbers (but can be solved with s.split("-",1)

Answers 6

In general with FOP ( functional oriented programming ) you can put it all in one liner and nest lambdas within one-liners but that is in general bad etiquette, since after 2 nesting function it all becomes quite unreadable.

The best way to approach this kind of issue is to split it up in several stages:

1: splitting string into tuple:

lst = ['b-3', 'a-2', 'c-4', 'd-2'] res = map( lambda str_x: tuple( str_x.split('-') ) , lst)    

2: sorting elements like you wished :

lst = ['b-3', 'a-2', 'c-4', 'd-2'] res = map( lambda str_x: tuple( str_x.split('-') ) , lst)   res = sorted( res, key=lambda x: ( int(x[1]), x[0] ) )  

Since we split the string into tuple it will return an map object that will be represented as list of tuples. So now the 3rd step is optional:

3: representing data as you inquired:

lst = ['b-3', 'a-2', 'c-4', 'd-2'] res = map( lambda str_x: tuple( str_x.split('-') ) , lst)   res = sorted( res, key=lambda x: ( int(x[1]), x[0] ) )  res = map( '-'.join, res )   

---

Now have in mind that lambda nesting could produce a more one-liner solution and that you can actually embed a non discrete nesting type of lambda like follows:

a = ['b-3', 'a-2', 'c-4', 'd-2'] resa = map( lambda x: x.split('-'), a) resa = map( lambda x: ( int(x[1]),x[0]) , a)  # resa can be written as this, but you must be sure about type you are passing to lambda  resa = map( lambda x: tuple( map( lambda y: int(y) is y.isdigit() else y , x.split('-') ) , a)   

But as you can see if contents of list a arent anything but 2 string types separated by '-' , lambda function will raise an error and you will have a bad time figuring what the hell is happening.

---

So in the end, i would like to show you several ways the 3rd step program could be written:

1:

lst = ['b-3', 'a-2', 'c-4', 'd-2'] res = map( '-'.join,\              sorted(\                    map( lambda str_x: tuple( str_x.split('-') ) , lst),\                        key=lambda x: ( int(x[1]), x[0] )\               )\          ) 

2:

lst = ['b-3', 'a-2', 'c-4', 'd-2'] res = map( '-'.join,\         sorted( map( lambda str_x: tuple( str_x.split('-') ) , lst),\                 key=lambda x: tuple( reversed( tuple(\                             map( lambda y: int(y) if y.isdigit() else y ,x  )\                         )))\             )\     )  # map isn't reversible 

3:

res = sorted( lst,\              key=lambda x:\                 tuple(reversed(\                     tuple( \                         map( lambda y: int(y) if y.isdigit() else y , x.split('-') )\                     )\                 ))\             ) 

So you can see how this all can get very complicated and incomprehensible. When reading my own or someone else's code i often love to see this version:

res = map( lambda str_x: tuple( str_x.split('-') ) , lst) # splitting string  res = sorted( res, key=lambda x: ( int(x[1]), x[0] ) ) # sorting for each element of splitted string res = map( '-'.join, res ) # rejoining string   

That is all from me. Have fun. I've tested all code in py 3.6.

---

PS. In general, you have 2 ways to approach lambda functions:

mult = lambda x: x*2   mu_add= lambda x: mult(x)+x #calling lambda from lambda 

This way is useful for typical FOP,where you have constant data , and you need to manipulate each element of that data. But if you need to resolve list,tuple,string,dict in lambda these kind of operations aren't very useful, since if any of those container/wrapper types is present the data type of elements inside containers becomes questionable. So we would need to go up a level of abstraction and determine how to manipulate data per its type.

mult_i = lambda x: x*2 if isinstance(x,int) else 2 # some ternary operator to make our life easier by putting if statement in lambda  

Now you can use another type of lambda function:

int_str = lambda x: ( lambda y: str(y) )(x)*x # a bit of complex, right?   # let me break it down.  #all this could be written as:  str_i = lambda x: str(x)  int_str = lambda x: str_i(x)*x  ## we can separate another function inside function with () ##because they can exclude interpreter to look at it first, then do the multiplication   # ( lambda x: str(x)) with this we've separated it as new definition of function   # ( lambda x: str(x) )(i) we called it and passed it i as argument.   

Some people call this type of syntax as nested lambdas, i call it indiscreet since you can see all.

And you can use recursive lambda assignment:

def rec_lambda( data, *arg_lambda ):       # filtering all parts of lambda functions parsed as arguments      arg_lambda = [ x for x in arg_lambda if type(x).__name__ == 'function' ]        # implementing first function in line     data = arg_lambda[0](data)        if arg_lambda[1:]: # if there are still elements in arg_lambda          return rec_lambda( data, *arg_lambda[1:] ) #call rec_lambda     else: # if arg_lambda is empty or []         return data # returns data    #where you can use it like this   a = rec_lambda( 'a', lambda x: x*2, str.upper, lambda x: (x,x), '-'.join)  >>> 'AA-AA'  

Answers 7

I think* if you are certain the format is consistently "[0]alphabet [1]dash" following indexes beyond [2:] will always be number, then you can replace split with slice, or you can use str.index('-')

sorted(lst, key=lambda x:(int(x[2:]),x[0]))  # str.index('-')  sorted(lst, key=lambda x:(int(x[x.index('-')+1 :]),x[0]))  

Read More

Are you able to use a custom Postgres comparison function for ORDER BY clauses?

In Python, I can write a sort comparison function which returns an item in the set {-1, 0, 1} and pass it to a sort function like so:

sorted(["some","data","with","a","nonconventional","sort"], custom_function) 

This code will sort the sequence according to the collation order I define in the function.

Can I do the equivalent in Postgres?

e.g.

SELECT widget FROM items ORDER BY custom_function(widget) 

Edit: Examples and/or pointers to documentation are welcome.

2 Answers

Answers 1

Yes you can, you can even create an functional index to speed up the sorting.

Edit: Simple example:

CREATE TABLE foo(     id serial primary key,     bar int ); -- create some data INSERT INTO foo(bar) SELECT i FROM generate_series(50,70) i; -- show the result SELECT * FROM foo;  CREATE OR REPLACE FUNCTION my_sort(int) RETURNS int  LANGUAGE sql  AS $$     SELECT $1 % 5; -- get the modulo (remainder) $$; -- lets sort! SELECT *, my_sort(bar) FROM foo ORDER BY my_sort(bar) ASC;  -- make an index as well: CREATE INDEX idx_my_sort ON foo ((my_sort(bar))); 

The manual is full of examples how to use your own functions, just start playing with it.

• SQL: http://www.postgresql.org/docs/current/static/xfunc-sql.html
• PL/pgSQL: http://www.postgresql.org/docs/current/static/plpgsql.html

Answers 2

You could do something like this

SELECT DISTINCT ON (interval_alias) *,   to_timestamp(floor((extract('epoch' FROM index.created_at) / 10)) * 10) AT   TIME ZONE 'UTC' AS interval_alias   FROM index   WHERE index.created_at >= '{start_date}'   AND index.created_at <= '{end_date}'   AND product = '{product_id}'   GROUP BY id, interval_alias ORDER BY interval_alias; 

Firstly you define the parameter that will be your ordering column with AS. It could be function or any SQL expression. Then set it to ORDER BY expression and you're done!

In my opinion, this is the smoothest way to do such an ordering.

Read More

Performance tips for finding unique permutation

TLDR: how to find multidimensional array permutation in php and how to optimize for bigger arrays?

This is continuation of this question: how to find multidimensional array permutation in php

we have script for sorting array, idea is to find unique permutation of array, rules to find this permutation are:

1. Input array contains set of arrays.
2. Each inner array contains unique elements.
3. Each inner array may have different length and different values.
4. Output array must contain exact same values.
5. Output inner array must have unique values on same key.
6. If there is no solution, wildcard ie.: null are allowed.
7. Wildcards can be duplicated on same key.
8. Solution should have as few wildcards as possible.
9. Algorithm should be able to handle array up to 30x30 in less than 180 s.

i have this solution so far:

function matrix_is_solved(array $matrix) {     foreach (array_keys(current($matrix)) as $offset) {         $column = array_filter($raw = array_column($matrix, $offset));         if (count($column) != count(array_unique($column))) return false;     }     return true; }  function matrix_generate_vectors(array $matrix) {     $vectors = [];     $columns = count(current($matrix));     $gen = function ($depth=0, $combo='') use (&$gen, &$vectors, $columns) {         if ($depth < $columns)              for ($i = 0; $i < $columns; $i++)                 $gen($depth + 1, $i . $combo);         else             $vectors[] = array_map('intval', str_split($combo));     };     $gen();     return $vectors; }  function matrix_rotate(array $matrix, array $vector) {    foreach ($matrix as $row => &$values) {        array_rotate($values, $vector[$row]);    }    return $matrix; }  function matrix_brute_solve(array $matrix) {     matrix_make_square($matrix);     foreach (matrix_generate_vectors($matrix) as $vector) {         $attempt = matrix_rotate($matrix, $vector);         if (matrix_is_solved($attempt))             return matrix_display($attempt);     }     echo 'No solution'; }  function array_rotate(array &$array, $offset) {     foreach (array_slice($array, 0, $offset) as $key => $val) {         unset($array[$key]);         $array[$key] = $val;     }     $array = array_values($array); }  function matrix_display(array $matrix = null) {     echo "[\n";     foreach ($matrix as $row => $inner) {         echo "  $row => ['" . implode("', '", $inner) . "']\n";     }     echo "]\n"; }  function matrix_make_square(array &$matrix) {     $pad = count(array_keys($matrix));     foreach ($matrix as &$row)         $row = array_pad($row, $pad, ''); }  $tests = [ [ ['X'], ['X'] ], [ ['X'], ['X'], ['X'] ], [ [ 'X', '' ], [ '', 'X' ] ], [ ['X', 'Y', 'Z'], ['X', 'Y'], ['X']], [ ['X', 'Y'], ['X', 'Y'], ['X', 'Y'] ] ]; array_map(function ($matrix) {     matrix_display($matrix);     echo "solved by:" . PHP_EOL;     matrix_brute_solve($matrix);     echo PHP_EOL; }, $tests); 

And this works perfectly on small array, but trouble is this iterating over all possibilities of array movements, and for array like 6x6 this is just too much to compute - O(nn) in both time and space!

3 Answers

Answers 1

The soluton is quite simple actually. You check the number of unique chars and that's the number of values in the output array. Below code will do what you want almost instantly.

/* HELPERS */  function ShowNice($output) {   //nice output:   echo '<pre>';   foreach($output as $key=>$val) {     echo '<br />' . str_pad($key,2," ",STR_PAD_LEFT) . ' => [';     $first = true;     foreach($val as $char) {       if (!$first) {         echo ', ';       }       echo "'".$char."'";       $first = false;     }     echo ']';   }   echo '</pre>'; }  function TestValid($output, $nullchar) {   $keys = count($output[0]);   for ($i=0;$i<$keys;$i++) {     $found = [];     foreach($output as $key=>$val) {       $char = $val[$i];       if ($char==$nullchar) {         continue;       }       if (array_key_exists($char, $found)) {         return false; //this char was found before       }       $found[$char] = true;     }   }    return true; }   $input = [   0 => ['X', 'Y', 'Z', 'I', 'J'],   1 => ['X', 'Y', 'Z', 'I'],   2 => ['X', 'Y', 'Z', 'I'],   3 => ['X', 'Y', 'Z', 'I'],   4 => ['X', 'Y', 'Z'],   5 => ['X', 'Y', 'Z'] ];  //generate large table $genLength = 30; //max double alphabet $innerLength = $genLength; $input2 = []; for($i=0;$i<$genLength;$i++) {   $inner = [];    if (rand(0, 1)==1) {     $innerLength--;   }    for($c=0;$c<$innerLength;$c++) {     $ascii = 65 + $c; //upper case     if ($ascii>90) {       $ascii += 6; //lower case     }     $inner[] = chr($ascii);   }   $input2[] = $inner; }   //generate large table with different keys $genLength = 10; //max double alphabet $innerLength = $genLength; $input3 = []; for($i=0;$i<$genLength;$i++) {   $inner = [];    if (rand(0, 1)==1) {     //comment o make same length inner arrays, but perhaps more distinct values     //$innerLength--;   }    $nr = 0;   for($c=0;$c<$innerLength;$c++) {     $ascii = 65 + $c + $nr; //upper case     if ($ascii>90) {       $ascii += 6; //lower case     }     $inner[] = chr($ascii);     //$inner[] = $c+$nr+1;      //increase nr?     if (rand(0, 2)==1) {       $nr++;     }    }   $input3[] = $inner; }   //generate table with numeric values, to show what happens $genLength = 10; //max double alphabet $innerLength = $genLength; $input4 = []; for($i=0;$i<$genLength;$i++) {   $inner = [];    for($c=0;$c<$innerLength;$c++) {     $inner[] = $c+1;   }   $input4[] = $inner; }   $input5 = [   0 => ['X', 'Y'],   1 => ['X', 'Y'],   2 => ['X', 'Y'], ];  $input6 = [   0 => ['X', 'Y', 'Z', 'I', 'J'],   1 => ['X', 'Y', 'Z', 'I'],   2 => ['X', 'Y', 'Z', 'I'],   3 => ['X', 'Y', 'Z', 'I'],   4 => ['X', 'Y', 'Z'] ];  /* ACTUAL CODE */  //$input = $input2;//test large table //$input = $input3;//test large unique table //$input = $input4;//test numeric //$input = $input5; //comment $input = $input6;  echo '<h2>Input</h2>'; ShowNice($input);   //find all distinct chars //find maxlength for any inner array  $distinct = []; $maxLength = 0; $minLength = -1; $rowCount = count($input); $flipped = []; $i = 1; foreach($input as $key=>$val) {   if ($maxLength<count($val)) {     $maxLength = count($val);   }   if ($minLength>count($val) || $minLength==-1) {     $minLength = count($val);   }   foreach($val as $char) {     if (!array_key_exists($char, $distinct)) {       $distinct[$char] = $i;       $i++;     }   }    $flipped[$key] = array_flip($val); }  //keep track of the count for actual chars $actualChars = $i-1; $nullchar = '_';     //add null values to distinct if ($minLength!=$maxLength) {   $char = '#'.$i.'#';   $distinct[$nullchar] = $i; //now it only gets add when a key is smaller, not if all are the same size   $i++; }  //if $distinct count is small then rowcount, we need more distinct $addForRowcount = (count($distinct)<$rowCount); while (count($distinct)<$rowCount) {   $char = '#'.$i.'#';   $distinct[$char] = $i;   $i++; }   //flip the distinct array to make the index the keys $distinct = array_flip($distinct);  $keys = count($distinct);  //create output $output = []; $start = 0; foreach($input as $key=>$val) {   $inner = [];   for ($i=1;$i<=$keys;$i++) {     $index = $start + $i;     if ($index>$keys) {         $index -= $keys;     }      if ($index>$actualChars) {       //just add the null char       $inner[] = $nullchar;     } else {       $char = $distinct[$index];        //check if the inner contains the char       if (!array_key_exists($char, $flipped[$key])) {         $char = $nullchar;       }        $inner[] = $char;     }    }   $output[] = $inner;   $start++; }   //UPDATE //at this point there can be a diagonal line with all wildcards. This can be removed to make it one part smaller //this wont happen when we added multiple distinct values because of the rowcount  //removing a diagonal line occasionally adds duplicates. //Keep the original and test if the new one is valid. If not, restore the original. $originalOutput = $output; $lastIndex = count($output[0])-1; for($i=$lastIndex;$i>=0;$i--) {   if ($output[0][$i]!=$nullchar) {     continue;//no wildcard   }   $found = true;   $moveleft=0;   foreach($output as $key=>$val){     if ($val[$i-$moveleft]!=$nullchar) {       $found = false;       break;//if it's not a NULL, we can stop checking     }     $moveleft++;   }    if ($found) {     //echo 'Found DIA';     //remove extra wildcards     $moveleft = 0;     foreach($output as $key=>$val){       $val[$i-$moveleft] = '*';       //comment below line to see which diagonal line is removed       unset($val[$i-$moveleft]);       $output[$key] = array_values($val); //make keys sequential again       $moveleft++;     }     break;   } }  echo '<h2>Original Output</h2>'; echo (TestValid($originalOutput, $nullchar)?'valid!!<br />':'INVALID!!<br />'); ShowNice($originalOutput);  echo '<h2>Output</h2>'; $valid = TestValid($output, $nullchar); echo ($valid?'valid!!<br />':'INVALID!!<br />'); ShowNice($output); if (!$valid) {   $output = $originalOutput; }  echo '<h2>Best result</h2>'; ShowNice($output); 

Result:

Input    0 => ['A', 'B', 'C', 'E', 'G', 'H', 'J', 'K', 'M']  1 => ['A', 'B', 'C', 'E', 'F', 'G', 'H', 'J']  2 => ['A', 'B', 'D', 'E', 'G', 'H', 'J', 'K']  3 => ['A', 'B', 'C', 'D', 'E', 'G', 'I']  4 => ['A', 'B', 'D', 'E', 'F', 'G']  5 => ['A', 'B', 'D', 'F', 'G', 'H']  6 => ['A', 'B', 'C', 'D', 'F', 'G']  7 => ['A', 'C', 'D', 'F', 'G', 'I']  8 => ['A', 'C', 'E', 'G', 'H', 'I']  9 => ['A', 'B', 'C', 'D', 'E', 'F']  Output    0 => ['A', 'B', 'C', 'E', 'G', 'H', 'J', 'K', 'M', '_', '_', '_', '_']  1 => ['B', 'C', 'E', 'G', 'H', 'J', '_', '_', 'F', '_', '_', '_', 'A']  2 => ['_', 'E', 'G', 'H', 'J', 'K', '_', '_', 'D', '_', '_', 'A', 'B']  3 => ['E', 'G', '_', '_', '_', '_', '_', 'D', 'I', '_', 'A', 'B', 'C']  4 => ['G', '_', '_', '_', '_', 'F', 'D', '_', '_', 'A', 'B', '_', 'E']  5 => ['H', '_', '_', '_', 'F', 'D', '_', '_', 'A', 'B', '_', '_', 'G']  6 => ['_', '_', '_', 'F', 'D', '_', '_', 'A', 'B', 'C', '_', 'G', '_']  7 => ['_', '_', 'F', 'D', 'I', '_', 'A', '_', 'C', '_', 'G', '_', '_']  8 => ['_', '_', '_', 'I', '_', 'A', '_', 'C', 'E', 'G', 'H', '_', '_']  9 => ['F', 'D', '_', '_', 'A', 'B', 'C', 'E', '_', '_', '_', '_', '_'] 

This is the result of the numbers table, as you can see it just moves the entire string one position to the right for each inner array. Then we just exchange the missing values for null chars.

Input    0 => ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10']  1 => ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10']  2 => ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10']  3 => ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10']  4 => ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10']  5 => ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10']  6 => ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10']  7 => ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10']  8 => ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10']  9 => ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10']  Output    0 => ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10']  1 => ['2', '3', '4', '5', '6', '7', '8', '9', '10', '1']  2 => ['3', '4', '5', '6', '7', '8', '9', '10', '1', '2']  3 => ['4', '5', '6', '7', '8', '9', '10', '1', '2', '3']  4 => ['5', '6', '7', '8', '9', '10', '1', '2', '3', '4']  5 => ['6', '7', '8', '9', '10', '1', '2', '3', '4', '5']  6 => ['7', '8', '9', '10', '1', '2', '3', '4', '5', '6']  7 => ['8', '9', '10', '1', '2', '3', '4', '5', '6', '7']  8 => ['9', '10', '1', '2', '3', '4', '5', '6', '7', '8']  9 => ['10', '1', '2', '3', '4', '5', '6', '7', '8', '9'] 

UPDATE

I updated the code because sometime we had an extra diagonal line of wildcards that's not needed. See the first result. We can unset the diagonal key, unless we need it because we had more rows then distinct values.

Result:

Input   0 => ['A', 'B', 'C', 'E', 'G', 'I', 'J', 'K', 'L']  1 => ['A', 'B', 'C', 'D', 'F', 'H', 'I', 'J']  2 => ['A', 'B', 'C', 'E', 'F', 'G', 'H', 'J']  3 => ['A', 'B', 'C', 'D', 'E', 'F', 'H']  4 => ['A', 'C', 'D', 'F', 'G', 'I']  5 => ['A', 'C', 'D', 'E', 'F', 'G']  6 => ['A', 'B', 'C', 'D', 'E']  7 => ['A', 'B', 'D', 'E']  8 => ['A', 'C', 'D']  9 => ['A', 'B']  Output   0 => ['A', 'B', 'C', 'E', 'G', 'I', 'J', 'K', 'L', '_', '_', '_']  1 => ['B', 'C', '_', '_', 'I', 'J', '_', '_', 'D', 'F', 'H', 'A']  2 => ['C', 'E', 'G', '_', 'J', '_', '_', '_', 'F', 'H', 'A', 'B']  3 => ['E', '_', '_', '_', '_', '_', 'D', 'F', 'H', 'A', 'B', 'C']  4 => ['G', 'I', '_', '_', '_', 'D', 'F', '_', 'A', '_', 'C', '_']  5 => ['_', '_', '_', '_', 'D', 'F', '_', 'A', '_', 'C', 'E', 'G']  6 => ['_', '_', '_', 'D', '_', '_', 'A', 'B', 'C', 'E', '_', '_']  7 => ['_', '_', 'D', '_', '_', 'A', 'B', '_', 'E', '_', '_', '_']  8 => ['_', 'D', '_', '_', 'A', '_', 'C', '_', '_', '_', '_', '_']  9 => ['_', '_', '_', 'A', 'B', '_', '_', '_', '_', '_', '_', '_'] 

UPDATE with your test:

Best result   0 => ['X', 'Y', 'Z', 'I', 'J']  1 => ['Y', 'Z', 'I', '_', 'X']  2 => ['Z', 'I', '_', 'X', 'Y']  3 => ['I', '_', 'X', 'Y', 'Z']  4 => ['_', 'X', 'Y', 'Z', '_'] 

Answers 2

what you should try to use is called a Power set which is:

from wikipedia in mathematics, the power set (or powerset) of any set S is the set of all subsets of S, including the empty set and S itself, variously denoted as P(S), 𝒫(S), ℘(S) (using the "Weierstrass p"), P(S), ℙ(S), or, identifying the powerset of S with the set of all functions from S to a given set of two elements, 2S.

if have a set of {a,b,c} it would give results of :

{{a,b,c},{a,b},{a,c},{b,c},{a},{b},{c}} 

a useful php library from github will give the right results you are looking for in above rules if not all rules applied you can also try to add filters on the results to get them right.

Answers 3

Based off the answer in the previous question you supplied this can be solved ( for that case ) way more elegantly using a few built in functions PHP has for array support. Which is probably the best of any language.

function solve($matrix){     $master = [];     $_matrix = [];     foreach($matrix as $key => $array){         $_matrix[$key] = array_combine($array,$array);         $master += $_matrix[$key];     }     $default = array_fill_keys($master, '');      $result = [];     foreach($_matrix as $array){         $result[] = array_values(array_merge($default, $array));     }     print_r($result); } 

Using their same tests

$tests = [     [ ['X'], ['X'] ],     [ ['X'], ['X'], ['X'] ],     [ [ 'X', '' ], [ '', 'X' ] ],     [ ['X', 'Y', 'Z'], ['X', 'Y'], ['X']],     [ ['X', 'Y'], ['X', 'Y'], ['X', 'Y'] ],     [ ['X', 'Y', 'Z'], ['X', 'Y', 'Z'], ['X', 'Y', 'Z'] ],     [ ['X', 'Y', 'Z', 'I', 'J'], ['X', 'Y', 'Z', 'I'], ['X', 'Y', 'Z', 'I'], ['X', 'Y', 'Z', 'I'], ['X', 'Y', 'Z'], ['X', 'Y', 'Z'] ], ]; array_map(function ($matrix) {     solve($matrix); }, $tests); 

This is what I get in comparison

[   0 => ['X', 'Y', 'Z', 'I', 'J'] //<- contains all unique values   1 => ['X', 'Y', 'Z', 'I']   2 => ['X', 'Y', 'Z', 'I']   3 => ['X', 'Y', 'Z', 'I']   4 => ['X', 'Y', 'Z']   5 => ['X', 'Y', 'Z'] ] Their Result: [   0 => ['', 'X', 'Y', 'Z', 'I', 'J'] //<- contains an extra '' empty value   1 => ['', '', 'X', 'Y', 'Z', 'I']   2 => ['I', '', '', 'X', 'Y', 'Z']   3 => ['Z', 'I', '', '', 'X', 'Y']   4 => ['Y', 'Z', '', '', '', 'X']   5 => ['X', 'Y', 'Z', '', '', ''] ] My Result [   0 => ['X', 'Y', 'Z', 'I', 'J']   1 => ['X', 'Y', 'Z', 'I', '']   2 => ['X', 'Y', 'Z', 'I', '']   3 => ['X', 'Y', 'Z', 'I', '']   4 => ['X', 'Y', 'Z','','']   5 => ['X', 'Y', 'Z','',''] ] 

You can test it here.

http://sandbox.onlinephpfunctions.com/code/86d0b4332963f95449df2e7d4d47fcd8224fe45d

I even timed it using microtime

mine 0.00013017654418945 milliseconds

theirs 0.10895299911499 milliseconds

Which is not really a surprise, as theirs is around 60 lines of code and 7 function calls. Mine is only 1 function 14 lines of code.

That said I don't know if the position of the values are important in the output. Nor exactly what you expect as the output extending that question.

The fact is they also lose the index position, just look at the second array in their result, 2 => ['I', '', '', 'X', 'Y', 'Z'] compared to the input 2 => ['X', 'Y', 'Z', 'I']. And I won't mention the extra '' in the output that probably doesn't belong there.

Maybe I'm missing something, lol, I don't typically do these math-y type things.

UPDATE if you want an explanation of how this works,

• array_combine($array,$array); creates an array with matched key => value, we abuse the fact that array keys are unique by nature. Like so ['X'=>'X','Y'=>'Y'...]
• then we build a "master" array with all the values in it and matched keys, one array to rule them all. The master array is limited in size to the max number or unique values because we are using the keys to eliminate duplicates.
• then we use array_fill_keys($master, ''); to sort of create a template of all the values. The keys of "master" are all the unique values in all of the inner arrays, so we fill it with our "wildcard" placeholder. In this case it looks like this ['X'=>'', 'Y'=>'', 'Z'=>'', 'I'=>'', 'J'=>'']
• then we merge the "modified" original array, also abusing the array keys for our advantage, by replacing the placeholders in the "templated" master array with the "modified" original array, because the keys match.
• last we strip the keys from the array using array_values

And we are left with each inner array "templated" by the master array but with the original values filled in and the missing ones empty.

Read More

Wordpress: Default sorting by column of custom post type

I have a custom post type called Contact, with custom fields like first name, surname, telephone number etc.

In the admin section they're sorted chronologically I think, but I need them to be sorted by surname by default.

I've read all the other solutions on here and none of them work, including:

function set_post_order_in_admin( $wp_query ) { global $pagenow;   if ( is_admin() && 'edit.php' == $pagenow && !isset($_GET['orderby'])) {     $wp_query->set( 'orderby', 'surname' );     $wp_query->set( 'order', 'ASC' );   } } add_filter('pre_get_posts', 'set_post_order_in_admin' ); 

But whatever field I try to sort by, nothing changes, except toggling ASC/DESC seems to change to reverse chronological ordering.

What am I doing wrong?

3 Answers

Answers 1

Refer below solutions,

function wpa84258_admin_posts_sort_last_name( $query ){     global $pagenow;     if( is_admin()         && 'edit.php' == $pagenow         && !isset( $_GET['orderby'] )         && !isset( $_GET['post_type'] ) ){             $query->set( 'meta_key', 'last_name' );             $query->set( 'orderby', 'meta_value' );             $query->set( 'order', 'ASC' );     } } add_action( 'pre_get_posts', 'wpa84258_admin_posts_sort_last_name' ); 

OR refer this solution

Answers 2

Replace

 $wp_query->set( 'orderby', 'surname' );  $wp_query->set( 'order', 'ASC' ); 

With

$query->set( 'meta_key', 'surname' ); // name of your post meta key $query->set( 'orderby',  'meta_value'); // meta_value since it is a string 

It may help

Answers 3

I am a drupal developer and haven't got a chance to play with WordPress. We had the same problem and this is how we fixed it.

Get your custom content type data (either WP default api or custom query), it would be an array of objects. Sort them using below function. return sorted array of posts. Not sure, in which hook you need to implement this in wordpress.

/**  *  Function to sort array by key  *  sortArrayByKey($yourArray,"name",true); //String sort (ascending order)  *  sortArrayByKey($yourArray,"name",true,false); //String sort (descending order)  *  sortArrayByKey($yourArray,"id"); //number sort (ascending order)  *  sortArrayByKey($yourArray,"count",false,false); //number sort (descending order)  */  function sortArrayByKey(&$array, $key, $string = false, $asc = true) {     if ($string) {         usort($array, function ($a, $b) use (&$key, &$asc) {             if ($asc) return strcmp(strtolower($a{$key}), strtolower($b{$key}));             else        return strcmp(strtolower($b{$key}), strtolower($a{$key}));         });     } else {         usort($array, function ($a, $b) use (&$key, &$asc) {             if ($a[$key] == $b{$key}) {                 return 0;             }             if ($asc) return ($a{$key} < $b{$key}) ? -1 : 1;             else     return ($a{$key} > $b{$key}) ? -1 : 1;          });     } } 

and then in your hook, you can call this function by

 return $this->sortArrayByKey($posts, "surname"); 

Function copied from this answer: https://stackoverflow.com/a/39872303/3086531

Read More

Sort by column within multi index level in pandas

I have a sorting request per example below.

Do i need to reset_index(), then sort() and then set_index() or is there a slick way to do this?

l = [[1,'A',99],[1,'B',102],[1,'C',105],[1,'D',97],[2,'A',19],[2,'B',14],[2,'C',10],[2,'D',17]] df = pd.DataFrame(l,columns = ['idx1','idx2','col1']) df.set_index(['idx1','idx2'],inplace=True)  # assume data has been received like this... print df             col1 idx1 idx2       1    A       99      B      102      C      105      D       97 2    A       19      B       14      C       10      D       17  # I'd like to sort descending on col1, partitioning within index level = 'idx2'             col1 idx1 idx2       1    C      105      B      102      A       99      D       97  2    A       19      D       17      B       14      C       10 

---

Thank you for the answer Note I change the data slightly:

l = [[1,'A',99],[1,'B',11],[1,'C',105],[1,'D',97],[2,'A',19],[2,'B',14],[2,'C',10],[2,'D',17]] df = pd.DataFrame(l,columns = ['idx1','idx2','col1']) df.set_index(['idx1','idx2'],inplace=True) df = df.sort_index(by='col1', ascending=False) 

however the output is

idx1 idx2       1    C      105      A       99      D       97 2    A       19      D       17      B       14 1    B       11 2    C       10 

i would have wanted it to be

idx1 idx2       1    C      105      A       99      D       97      B       11  2    A       19      D       17      B       14      C       10 

3 Answers

Answers 1

you can use sort_index:

 df.sort_index(by='col1', ascending=False) 

This outputs:

             col1 idx1    idx2     1       C    105         B    102         A    99         D    97 2       A    19         D    17         B    14         C    10 

Answers 2

You need DataFrame.reset_index, DataFrame.sort_values and DataFrame.set_index::

l = [[1,'A',99],[1,'B',11],[1,'C',105],[1,'D',97],      [2,'A',19],[2,'B',14],[2,'C',10],[2,'D',17]] df = pd.DataFrame(l,columns = ['idx1','idx2','col1']) df.set_index(['idx1','idx2'],inplace=True) print (df)            col1 idx1 idx2       1    A       99      B       11      C      105      D       97 2    A       19      B       14      C       10      D       17  df = df.reset_index() \        .sort_values(['idx1','col1'], ascending=[True,False]) \        .set_index(['idx1','idx2']) print (df)            col1 idx1 idx2       1    C      105      A       99      D       97      B       11 2    A       19      D       17      B       14      C       10 

Answers 3

This first sorts by the desired column, the resorts on the idx1 MultiIndex level only and works in up to date pandas versions that deprecate the by kwarg.

df.sort_values('col1', ascending=False).sort_index(level='idx1', sort_remaining=False) 

Output:

             col1 idx1    idx2     1       C    105         B    102         A    99         D    97 2       A    19         D    17         B    14         C    10 

Read More

Sort by array's last element mongodb

I was trying to sort documents by last interaction. meta_data.access_times is an array that update every time when user interacts and new date object append to the last element of the array. Is there any way to sort by array's last element?

Attempt 1 :

private Aggregation makeQuery(String userId) {      return newAggregation(           match(Criteria.where("user_id").is(userId)),           sort(Sort.Direction.DESC, "$meta_data.access_times"),           group(Fields.fields().and("first_name", "$meta_data.user_data.first_name").and("last_name", "$meta_data.user_data.last_name").and("profile_pic", "$meta_data.user_data.profile_pic").and("user_id", "$user_id").and("access_times", "$meta_data.access_times"))       );     } 

Attempt 2 :

 private Aggregation makeQuery(String userId) {         return newAggregation(             match(Criteria.where("user_id").is(user_id)),             group(Fields.fields().and("first_name", "$meta_data.user_data.first_name").and("last_name", "$meta_data.user_data.last_name").and("profile_pic", "$meta_data.user_data.profile_pic").and("user_id", "$user_id")).max("$meta_data.access_times").as("access_time"),             sort(Sort.Direction.DESC, "access_time")         );     } 

sample meta_data array in document

"meta_data" : { "access_times" : [              ISODate("2017-06-20T14:04:14.910Z"),              ISODate("2017-06-22T06:27:32.210Z"),              ISODate("2017-06-22T06:27:35.326Z"),              ISODate("2017-06-22T06:31:28.048Z"),              ISODate("2017-06-22T06:36:19.664Z"),              ISODate("2017-06-22T06:37:00.164Z")         ] } 

2 Answers

Answers 1

I solves the problem by using $unwind operation.

 private Aggregation makeQuery(String userId) {         return newAggregation(             match(Criteria.where("user_id").is(userId)),             unwind("$meta_data.access_times"),             group(Fields.fields().and("first_name", "$meta_data.user_data.first_name").and("last_name", "$meta_data.user_data.last_name").and("profile_pic", "$meta_data.user_data.profile_pic").and("user_id", "$user_id")).max("$meta_data.access_times").as("access_time"),             sort(Sort.Direction.DESC, "access_time")         );     } 

Answers 2

When you don't know if the element to Push is Ordered or not (for example, an User that is pushing him Score...) you can use $push and $sort in order to have an ordered array, then you can just sort by "find({userId:yourUseId}.sort("metadata.access_time.0":-1).

This solution suppose your array are Ordered with $sort at creation/update time: LINK

When you are sure that the Push don't need a sort (for example you are Pushing a Access_Date for that User) you can $push and void $sort by using $operator (tnx Erdenezul). LINK

In theory you don't need an Index on the Array "access_time" if the find() is fetching only fews documents. Otherwise you can just add an index with {"metadata.access_time.0": -1}.

Good Luck!

Read More

Sort array values based on parent/child relationship

I am trying to sort an array to ensure that the parent of any item always exists before it in the array. For example:

Array (     [0] => Array         (             [0] => 207306             [1] => Bob             [2] =>          )      [1] => Array         (             [0] => 199730             [1] => Sam             [2] => 199714         )      [2] => Array         (             [0] => 199728             [1] => Simon             [2] => 207306         )      [3] => Array         (             [0] => 199714             [1] => John             [2] => 207306         )      [4] => Array         (             [0] => 199716             [1] => Tom             [2] => 199718         )      [5] => Array         (             [0] => 199718             [1] => Phillip             [2] => 207306         )      [6] => Array         (             [0] => 199720             [1] => James             [2] => 207306         )  ) 

In the above array this "fails" as [1][2] (Sam) does not yet exist and nor does [4][2] (Tom).

The correct output would be as, in this case, as both Sam and Tom's parents already exist before they appear in the array:

Array (     [0] => Array         (             [0] => 207306             [1] => Bob             [2] =>          )      [1] => Array         (             [0] => 199714             [1] => John             [2] => 207306         )       [2] => Array         (             [0] => 199730             [1] => Sam             [2] => 199714         )      [3] => Array         (             [0] => 199728             [1] => Simon             [2] => 207306         )       [4] => Array         (             [0] => 199718             [1] => Phillip             [2] => 207306         )       [5] => Array         (             [0] => 199716             [1] => Tom             [2] => 199718         )      [6] => Array         (             [0] => 199720             [1] => James             [2] => 207306         )  ) 

I found an answer http://stackoverflow.com/a/12961400/1278201 which was very close but it only seems to go one level deep (i.e. there is only ever one parent) whereas in my case there could be 1 or 10 levels deep in the hierarchy.

How do I sort the array so no value can appear unless its parent already exists before it?

6 Answers

Answers 1

This will trivially order the array (in O(n)) putting first all those with no parent, then these whose parent is already in the array, iteratively, until there's no children having the current element as parent.

# map the children by parent $parents = ['' => []]; foreach ($array as $val) {     $parents[$val[2]][] = $val; } # start with those with no parent $sorted = $parents['']; # add the children the current nodes are parent of until the array is empty foreach ($sorted as &$val) {     if (isset($parents[$val[2]])) {         foreach ($parents[$val[2]] as $next) {             $sorted[] = $next;         }     } } 

This code requires PHP 7, it may not work in some cases under PHP 5. - for PHP 5 compatibility you will have to swap the foreach ($sorted as &$val) with for ($val = reset($sorted); $val; $val = next($sorted)):

# a bit slower loop which works in all versions for ($val = reset($sorted); $val; $val = next($sorted)) {     if (isset($parents[$val[2]])) {         foreach ($parents[$val[2]] as $next) {             $sorted[] = $next;         }     } } 

Answers 2

What about this approach:

Create an empty array result.

Loop over your array and only take the items out of it where [2] is empty and insert them into result.

When this Loop is done you use a foreach-Loop inside a while-loop. With the foreach-Loop you take every item out of your array where [2] is already part of result. And you do this as long as your array contains anything.

$result = array(); $result[''] = 'root';  while(!empty($yourArray)){   foreach($yourArray as $i=>$value){     if(isset($result[$value[2]])){       // use the next line only to show old order       $value['oldIndex'] = $i;       $result[$value[0]] = $value;       unset($yourArray[$i]);     }   } }  unset($result['']); 

PS: You may run into trouble by removing parts of an array while walking over it. If you do so ... try to solve this :)

PPS: Think about a break condition if your array have an unsolved loop or a child without an parent.

Answers 3

I checked this works in PHP 5.6 and PHP 7

Sample array:

$array = Array(0 => Array(         0 => 207306,         1 => 'Bob',         2 => '',     ),     1 => Array         (         0 => 199730,         1 => 'Sam',         2 => 199714,     ),     2 => Array         (         0 => 199728,         1 => 'Simon',         2 => 207306,     ),     3 => Array         (         0 => 199714,         1 => 'John',         2 => 207306,     ),     4 => Array         (         0 => 199716,         1 => 'Tom',         2 => 199718,     ),     5 => Array         (         0 => 199718,         1 => 'Phillip',         2 => 207306,     ),     6 => Array         (         0 => 199720,         1 => 'James',         2 => 207306,     ), );     echo "<pre>"; $emp = array();  //form the array with parent and child foreach ($array as $val) {     $manager = ($val[2] == '') ? 0 : $val[2];     $exist = array_search_key($val[2], $emp);     if ($exist)         $emp[$exist[0]][$val[0]] = $val;     else     //print_R(array_search_key(199714,$emp));         $emp[$manager][$val[0]] = $val; }  $u_emp = $emp[0]; unset($emp[0]);  //associate the correct child/emp after the manager foreach ($emp as $k => $val) {     $exist = array_search_key($k, $u_emp);     $pos = array_search($k, array_keys($u_emp));      $u_emp = array_slice($u_emp, 0, $pos+1, true) +             $val +             array_slice($u_emp, $pos-1, count($u_emp) - 1, true);  } print_R($u_emp); //print the final result  // key search function from the array function array_search_key($needle_key, $array, $parent = array()) {     foreach ($array AS $key => $value) {         $parent = array();         if ($key == $needle_key)             return $parent;         if (is_array($value)) {             array_push($parent, $key);             if (($result = array_search_key($needle_key, $value, $parent)) !== false)                 return $parent;         }     }     return false; } 

Answers 4

you can use your array in variable $arr and use this code it will give you required output.

function check($a, $b) {        return ($a[0] == $b[2]) ? -1 : 1; }   uasort($arr, 'check'); echo '<pre>'; print_r(array_values($arr)); echo '</pre>'; 

Answers 5

Find the below code that might be helpful.So, your output is stored in $sortedarray.

$a=array(array(207306,'Bob',''), array (199730,'Sam',199714), array(199728,'Simon',207306), array(199714,'John',207306), array(199716,'Tom',199718), array(199718,'Phillip',207306), array(199720,'James',207306));  $sortedarray=$a; foreach($a as $key=>$value){     $checkvalue=$value[2];     $checkkey=$key; foreach($a as $key2=>$value2){     if($key<$key2){             if ($value2[0]===$checkvalue){                 $sortedarray[$key]=$value2;                 $sortedarray[$key2]=$value;         }else{             }     }   }  }   print_r($sortedarray); 

Answers 6

I have two different version for you.

a) Using a "walk the tree" approach with recursion and references to minimize memory consumption

$data = [     [207306,'Bob',''], [199730,'Sam',199714],     [199728,'Simon',207306], [199714,'John',207306],     [199716, 'Tom',199718], [199718,'Phillip',207306],     [199720,'James',207306] ];  $list = []; generateList($data, '', $list);  var_dump($list);  function generateList($data, $id, &$list) {     foreach($data as $d) {         if($d[2] == $id) {                       $list[] = $d; // Child found, add it to list                         generateList($data, $d[0], $list); // Now search for childs of this child         }     } } 

b) Using phps built in uusort()function (seems only to work up to php 5.x and not with php7+)

$data = [     [207306,'Bob',''], [199730,'Sam',199714],     [199728,'Simon',207306], [199714,'John',207306],     [199716, 'Tom',199718], [199718,'Phillip',207306],     [199720,'James',207306] ];  usort($data, 'cmp');  var_dump($data);  function cmp($a, $b) {       if($a[2] == '' || $a[0] == $b[2]) return -1; //$a is root element or $b is child of $a     if($b[2] == '' || $b[0] == $a[2]) return 1; //$b is root element or $a is child of $b     return 0; // both elements have no direct relation } 

Read More

Getting Product Collection Sorted by minimum price

Summary of Work Environment

I am working on a website where we have customer and dealers both. Each Dealer can have their own price for a product.

Production collection data is having another duplicate record (CLONING PRODUCT) for each product having price of that seller. For example if master catalog have IPHONE 6S . than 5 dealers who deal in Iphone 6s can have their own prices. Cloning product creates a new product ID related to Seller ID

Requirement

I need to get the category wise product listing having lowest price of dealer. Also need to sort that listing according to lowest price.

what I tried

Currently I can list out all the products having lowest price according to category.

$productCollection = Mage::getResourceModel('catalog/product_collection')                     ->addAttributeToSelect('sellingprice')                     ->setStoreId($storeId)                     ->joinField('category_id', 'catalog/category_product', 'category_id', 'product_id=entity_id', null, 'left')                     ->addAttributeToFilter('category_id', array('in' => $_POST['category_id']))                     ->addAttributeToFilter('status', array('eq' => 1))                     ->addAttributeToFilter('dis_continue', array('eq' => 0));    $productCollection->addAttributeToFilter('seller_id', array('in' => $seller_list));  $productCollection->addExpressionAttributeToSelect(                     'lowest_price', 'IF(({{special_from_date}}<=now() AND {{special_to_date}}>=now() OR {{special_from_date}} IS NULL AND {{special_price}}>0),{{special_price}},IF({{sellingprice}}>0,{{sellingprice}},{{price}}))', array('special_from_date', 'special_to_date', 'special_price', 'sellingprice', 'price'));   $productCollection->getSelect()->columns('MIN(IF((IF(at_special_from_date.value_id > 0, at_special_from_date.value, at_special_from_date_default.value)<=now() AND IF(at_special_to_date.value_id > 0, at_special_to_date.value, at_special_to_date_default.value)>=now() OR IF(at_special_from_date.value_id > 0, at_special_from_date.value, at_special_from_date_default.value) IS NULL AND at_special_price.value>0),at_special_price.value,IF(at_sellingprice.value>0,at_sellingprice.value,at_price.value))) as l_price')->group('product_name'); 

I find out lowest of selling price , special price , mrp of a dealer.

Using Group By which groups all the data by Product Name , get MINIMUM of Lowest Price , SORTING that according to LOWEST Price.

PROBLEM

As I explained that I am Using GROUP BY Name so that I can have unique products but I am not able to get the PRODUCT ID of associated seller who is having lowest price. I need to get the Seller ID Of having LOWEST PRICE

GROUP BY always Returns the first ROW , but MIN() function gives the lowest of price. First ROW do not have the associated PRODUCT ID of lowest price.....

EDIT - MYSQL QUERY

SELECT `e`.*, `at_category_id`.`category_id`, IF(   at_status.value_id > 0,   at_status.value,   at_status_default.value ) AS `status`, `at_dis_continue`.`value` AS `dis_continue`, `at_seller_id`.`value` AS `seller_id`, `at_popular_product`.`value` AS `popular_product`, IF(   at_special_from_date.value_id > 0,   at_special_from_date.value,   at_special_from_date_default.value ) AS `special_from_date`, IF(   at_special_to_date.value_id > 0,   at_special_to_date.value,   at_special_to_date_default.value ) AS `special_to_date`, `at_special_price`.`value` AS `special_price`, `at_sellingprice`.`value` AS `sellingprice`, `at_price`.`value` AS `price`, IF(   (     IF(       at_special_from_date.value_id > 0,       at_special_from_date.value,       at_special_from_date_default.value     ) <= NOW() AND IF(       at_special_to_date.value_id > 0,       at_special_to_date.value,       at_special_to_date_default.value     ) >= NOW() OR IF(       at_special_from_date.value_id > 0,       at_special_from_date.value,       at_special_from_date_default.value     ) IS NULL AND at_special_price.value > 0   ),   at_special_price.value,   IF(     at_sellingprice.value > 0,     at_sellingprice.value,     at_price.value   ) ) AS `lowest_price`, `at_name`.`value` AS `name`, `at_name`.`value` AS `product_name`, MIN(   IF(     (       IF(         at_special_from_date.value_id > 0,         at_special_from_date.value,         at_special_from_date_default.value       ) <= NOW() AND IF(         at_special_to_date.value_id > 0,         at_special_to_date.value,         at_special_to_date_default.value       ) >= NOW() OR IF(         at_special_from_date.value_id > 0,         at_special_from_date.value,         at_special_from_date_default.value       ) IS NULL AND at_special_price.value > 0     ),     at_special_price.value,     IF(       at_sellingprice.value > 0,       at_sellingprice.value,       at_price.value     )   ) ) AS `l_price` FROM   `catalog_product_entity` AS `e` LEFT JOIN   `catalog_category_product` AS `at_category_id` ON(     at_category_id.`product_id` = e.entity_id   ) INNER JOIN   `catalog_product_entity_int` AS `at_status_default` ON(     `at_status_default`.`entity_id` = `e`.`entity_id`   ) AND(     `at_status_default`.`attribute_id` = '96'   ) AND `at_status_default`.`store_id` = 0 LEFT JOIN   `catalog_product_entity_int` AS `at_status` ON(     `at_status`.`entity_id` = `e`.`entity_id`   ) AND(`at_status`.`attribute_id` = '96') AND(`at_status`.`store_id` = 1) INNER JOIN   `catalog_product_entity_int` AS `at_dis_continue` ON(     `at_dis_continue`.`entity_id` = `e`.`entity_id`   ) AND(     `at_dis_continue`.`attribute_id` = '261'   ) AND(`at_dis_continue`.`store_id` = 0) INNER JOIN   `catalog_product_entity_varchar` AS `at_seller_id` ON(     `at_seller_id`.`entity_id` = `e`.`entity_id`   ) AND(     `at_seller_id`.`attribute_id` = '134'   ) AND(`at_seller_id`.`store_id` = 0) INNER JOIN   `catalog_product_entity_varchar` AS `at_popular_product` ON(     `at_popular_product`.`entity_id` = `e`.`entity_id`   ) AND(     `at_popular_product`.`attribute_id` = '1078'   ) AND(     `at_popular_product`.`store_id` = 0   ) LEFT JOIN   `catalog_product_entity_datetime` AS `at_special_from_date_default` ON(     `at_special_from_date_default`.`entity_id` = `e`.`entity_id`   ) AND(     `at_special_from_date_default`.`attribute_id` = '77'   ) AND `at_special_from_date_default`.`store_id` = 0 LEFT JOIN   `catalog_product_entity_datetime` AS `at_special_from_date` ON(     `at_special_from_date`.`entity_id` = `e`.`entity_id`   ) AND(     `at_special_from_date`.`attribute_id` = '77'   ) AND(     `at_special_from_date`.`store_id` = 1   ) LEFT JOIN   `catalog_product_entity_datetime` AS `at_special_to_date_default` ON(     `at_special_to_date_default`.`entity_id` = `e`.`entity_id`   ) AND(     `at_special_to_date_default`.`attribute_id` = '78'   ) AND `at_special_to_date_default`.`store_id` = 0 LEFT JOIN   `catalog_product_entity_datetime` AS `at_special_to_date` ON(     `at_special_to_date`.`entity_id` = `e`.`entity_id`   ) AND(     `at_special_to_date`.`attribute_id` = '78'   ) AND(     `at_special_to_date`.`store_id` = 1   ) LEFT JOIN   `catalog_product_entity_decimal` AS `at_special_price` ON(     `at_special_price`.`entity_id` = `e`.`entity_id`   ) AND(     `at_special_price`.`attribute_id` = '76'   ) AND(`at_special_price`.`store_id` = 0) LEFT JOIN   `catalog_product_entity_decimal` AS `at_sellingprice` ON(     `at_sellingprice`.`entity_id` = `e`.`entity_id`   ) AND(     `at_sellingprice`.`attribute_id` = '143'   ) AND(`at_sellingprice`.`store_id` = 0) LEFT JOIN   `catalog_product_entity_decimal` AS `at_price` ON(     `at_price`.`entity_id` = `e`.`entity_id`   ) AND(`at_price`.`attribute_id` = '75') AND(`at_price`.`store_id` = 0) LEFT JOIN   `catalog_product_entity_varchar` AS `at_name` ON(     `at_name`.`entity_id` = `e`.`entity_id`   ) AND(`at_name`.`attribute_id` = '71') AND(`at_name`.`store_id` = 0) WHERE   (     at_category_id.category_id IN('119')   ) AND(     IF(       at_status.value_id > 0,       at_status.value,       at_status_default.value     ) = 1   ) AND(at_dis_continue.value = 0) AND(at_seller_id.value IN('1065')) AND(     at_popular_product.value IN('Yes',     'No')   ) GROUP BY   `product_name` 

Please help.

1 Answers

Answers 1

I'm afraid I'm not familiar enough with Magento itself to help directly with your code, but, more generally speaking, this is a common question when it comes to SQL SELECT queries.

GROUP BY

Firstly, an important clarification: When using GROUP BY, any fields in the SELECT part of the query not included in the GROUP BY clause itself may not be legal. The outcome depends on your server version and/or the ONLY_FULL_GROUP_BY SQL mode.

More importantly, assuming your server/configuration supports it, selecting fields not included in the GROUP BY clause means you get a value from an arbitrary row in the group, not the first row. From the MySQL Handling of GROUP BY page in the MySQL documentation:

In this case, the server is free to choose any value from each group, so unless they are the same, the values chosen are indeterminate, which is probably not what you want.

Selecting specific rows within groups

One way of achieving the behaviour you're looking for that has always worked well for me is by using counters and sub-queries to order and filter your sub-groups. This gives you a greater level of control than a GROUP BY (although you do make some performance sacrifices):

SELECT @num := IF(products_name=@last_products_name, @num + 1, 1) b, (@last_products_name := products_name) AS last_pname, t1.* FROM (     SELECT p.products_id, p.products_name, p.selling_price     FROM products p     WHERE p.category_id = 123     ORDER BY p.products_name,     p.selling_price ASC ) t1, (SELECT @num := 0, @last_products_name := 0) d HAVING b=1; 

To understand more clearly how this works, run the query without the HAVING clause. You get a result like this:

+------+------------+-------------+---------------+---------------+ | b    | last_pname | products_id | products_name | selling_price | +------+------------+-------------+---------------+---------------+ |    1 | Bar        |           8 | Bar           |          5.00 | |    2 | Bar        |           2 | Bar           |         12.00 | |    3 | Bar        |           4 | Bar           |         14.00 | |    1 | Fizz       |           3 | Fizz          |         30.00 | |    2 | Fizz       |           5 | Fizz          |         70.00 | |    3 | Fizz       |           7 | Fizz          |        100.00 | |    1 | Foo        |           1 | Foo           |         10.00 | |    2 | Foo        |           6 | Foo           |         18.00 | +------+------------+-------------+---------------+---------------+ 

The b column shows the value of the @num variable, which is incremented for each row in a group of identically named products, and reset each time the product name in the current row is not equal to the name of the last one. Adding the HAVING b=1 clause means we only get the cheapest product in each group.

A potential gotcha when using ORDER BY in sub-queries!

When I last used MySQL, the above solution would work (and I imagine that is still true now). However, this is not actually standard SQL behaviour. Database servers which adhere more strictly to the standard (such as MariaDB) will ignore an ORDER BY clause contained within a sub-query, unless the sub-query also features a LIMIT clause. Therefore, if you are using MariaDB, you need to force the server to honour the ORDER BY by including a LIMIT. A technique I have used before (as described in a comment on the previous link) is to specify a very large LIMIT value:

SELECT @num := IF(products_name=@last_products_name, @num + 1, 1) b, (@last_products_name := products_name) AS last_pname, t1.* FROM (     SELECT p.products_id, p.products_name, p.selling_price     FROM products p     WHERE p.category_id = 123     ORDER BY p.products_name,     p.selling_price ASC     LIMIT 18446744073709551615 -- LIMIT clause forces sub-query ORDER BY ) t1, (SELECT @num := 0, @last_products_name := 0) d HAVING b=1; 

I hope that helps.

Read More

Sorting a data stream before writing to file in nodejs

I have an input file which may potentially contain upto 1M records and each record would look like this

field 1 field 2 field3 \n

I want to read this input file and sort it based on field3 before writing it to another file.

here is what I have so far

var fs = require('fs'),     readline = require('readline'),     stream = require('stream');  var start = Date.now();  var outstream = new stream; outstream.readable = true; outstream.writable = true;  var rl = readline.createInterface({     input: fs.createReadStream('cross.txt'),     output: outstream,     terminal: false });  rl.on('line', function(line) {     //var tmp = line.split("\t").reverse().join('\t') + '\n';     //fs.appendFileSync("op_rev.txt", tmp );     // this logic to reverse and then sort is too slow });  rl.on('close', function() {     var closetime = Date.now();     console.log('Read entirefile. ', (closetime - start)/1000, ' secs'); }); 

I am basically stuck at this point, all I have is the ability to read from one file and write to another, is there a way to efficiently sort this data before writing it

4 Answers

Answers 1

DB and sort-stream are fine solutions, but DB might be an overkill and I think sort-stream eventually just sorts the entire file in an in-memory array (on through end callback), so I think performance will be roughly the same, comparing to the original solution.
(but I haven't ran any benchmarks, so I might be wrong).

So, just for the hack of it, I'll throw in another solution :)

---

EDIT: I was curious to see how big a difference this will be, so I ran some benchmarks.

Results were surprising even to me, turns out sort -k3,3 solution is better by far, x10 times faster then the original solution (a simple array sort), while nedb and sort-stream solutions are at least x18 times slower than the original solution (i.e. at least x180 times slower than sort -k3,3).

(See benchmark results below)

---

If on a *nix machine (Unix, Linux, Mac, ...) you can simply use
sort -k 3,3 yourInputFile > op_rev.txt and let the OS do the sorting for you.
You'll probably get better performance, since sorting is done natively.

Or, if you want to process the sorted output in Node:

var util = require('util'),     spawn = require('child_process').spawn,     sort = spawn('sort', ['-k3,3', './test.tsv']);  sort.stdout.on('data', function (data) {     // process data     data.toString()         .split('\n')         .map(line => line.split("\t"))         .forEach(record => console.info(`Record: ${record}`)); });  sort.on('exit', function (code) {     if (code) {         // handle error     }      console.log('Done'); });  // optional sort.stderr.on('data', function (data) {     // handle error...     console.log('stderr: ' + data); }); 

Hope this helps :)

---

EDIT: Adding some benchmark details.

I was curious to see how big a difference this will be, so I ran some benchmarks.

Here are the results (running on a MacBook Pro):

• sort1 uses a straightforward approach, sorting the records in an in-memory array.
  Avg time: 35.6s (baseline)

• sort2 uses sort-stream, as suggested by Joe Krill.
  Avg time: 11.1m (about x18.7 times slower)
  (I wonder why. I didn't dig in.)

• sort3 uses nedb, as suggested by Tamas Hegedus.
  Time: about 16m (about x27 times slower)

• sort4 only sorts by executing sort -k 3,3 input.txt > out4.txt in a terminal
  Avg time: 1.2s (about x30 times faster)

• sort5 uses sort -k3,3, and process the response sent to stdout
  Avg time: 3.65s (about x9.7 times faster)

Answers 2

You can take advantage of streams for something like this. There's a few NPM modules that will be helpful -- first include them by running

npm install sort-stream csv-parse stream-transform 

from the command line.

Then:

var fs = require('fs'); var sort = require('sort-stream'); var parse = require('csv-parse'); var transform = require('stream-transform');  // Create a readble stream from the input file. fs.createReadStream('./cross.txt')   // Use `csv-parse` to parse the input using a tab character (\t) as the    // delimiter. This produces a record for each row which is an array of    // field values.   .pipe(parse({     delimiter: '\t'   }))   // Use `sort-stream` to sort the parsed records on the third field.    .pipe(sort(function (a, b) {     return a[2].localeCompare(b[2]);   }))   // Use `stream-transform` to transform each record (an array of fields) into    // a single tab-delimited string to be output to our destination text file.   .pipe(transform(function(row) {     return row.join('\t') + '\r';   }))   // And finally, output those strings to our destination file.   .pipe(fs.createWriteStream('./cross_sorted.txt')); 

Answers 3

You have two options, depending on how much data is being processed. (1M record count with 3 columns doesn't say much about the amount of actual data)

Load the data in memory, sort in place

var lines = []; rl.on('line', function(line) {     lines.push(line.split("\t").reverse()); });  rl.on('close', function() {     lines.sort(function(a, b) { return compare(a[0], b[0]); });      // write however you want     fs.writeFileSync(         fileName,         lines.map(function(x) { return x.join("\t"); }).join("\n")     );     function compare(a, b) {         if (a < b) return -1;         if (a > b) return 1;         return 0;     } }); 

Load the data in a persistent database, read ordered

Using a database engine of your choice (for example nedb, a pure javascript db for nodejs)

EDIT: It seems that NeDB keeps the whole database in memory, the file is only a persistent copy of the data. We'll have to search for another implementation. TingoDB looks promising.

// This code is only to give an idea, not tested in any way  var Datastore = require('nedb'); var db = new Datastore({     filename: 'path/to/temp/datafile',     autoload: true });  rl.on('line', function(line) {     var tmp = line.split("\t").reverse();     db.insert({         field0: tmp[0],         field1: tmp[1],         field2: tmp[2]     }); });  rl.on('close', function() {     var cursor = db.find({})             .sort({ field0: 1 }); // sort by field0, ascending     var PAGE_SIZE = 1000;     paginate(0);     function paginate(i) {         cursor.skip(i).take(PAGE_SIZE).exec(function(err, docs) {             // handle errors              var tmp = docs.map(function(o) {                 return o.field0 + "\t" + o.field1 + "\t" + o.field2 + "\n";             });             fs.appendFileSync("op_rev.txt", tmp.join(""));             if (docs.length >= PAGE_SIZE) {                 paginate(i + PAGE_SIZE);             } else {                 // cleanup temp database             }         });     } }); 

Answers 4

i had quite similar issue, needed to perform an external sort.

I figured out, after waste a few time on it that i could load up the data on a database and then query out the desired data from it.

It not even matter if the inserts aren't ordered, as long as my query result could be.

Hope it can work for you too.

In order to insert your data on a database, there are plenty of tools on node to perform such task. I have this pet project which does a similar job.

I'm also sure that if you search the subject, you'll find much more info.

Good luck.

Read More