Showing posts with label string. Show all posts
Showing posts with label string. Show all posts

Friday, September 21, 2018

Detect differences between two strings

Leave a Comment

I have 2 strings

string a = "foo bar"; string b = "bar foo"; 

and I want to detect the changes from a to b. What characters do I have to change, to get from a to b?

I think there must be a iteration over each character and detect if it was added, removed or remained equal. So this is my exprected result

'f' Remove 'o' Remove 'o' Remove ' ' Remove 'b' Equal 'a' Equal 'r' Equal ' ' Add 'f' Add 'o' Add 'o' Add 

class and enum for the result:

public enum Operation { Add,Equal,Remove }; public class Difference {     public Operation op { get; set; }     public char c { get; set; } } 

Here is my solution but the "Remove" case is not clear to me how the code has to look like

public static List<Difference> CalculateDifferences(string left, string right) {     int count = 0;     List<Difference> result = new List<Difference>();     foreach (char ch in left)     {         int index = right.IndexOf(ch, count);         if (index == count)         {             count++;             result.Add(new Difference() { c = ch, op = Operation.Equal });         }         else if (index > count)         {             string add = right.Substring(count, index - count);             result.AddRange(add.Select(x => new Difference() { c = x, op = Operation.Add }));             count += add.Length;         }         else         {             //Remove?         }     }     return result; } 

How does the code have to look like for removed characters?


Update - added a few more examples

example 1:

string a = "foobar"; string b = "fooar"; 

expected result:

'f' Equal 'o' Equal 'o' Equal 'b' Remove 'a' Equal 'r' Equal 

example 2:

string a = "asdfghjk"; string b = "wsedrftr"; 

expected result:

'a' Remove 'w' Add 's' Equal 'e' Add 'd' Equal 'r' Add 'f' Equal 'g' Remove 'h' Remove 'j' Remove 'k' Remove 't' Add 'r' Add 

Update:

Here is a comparison between Dmitry's and ingen's answer: https://dotnetfiddle.net/MJQDAO

5 Answers

Answers 1

You are looking for (minimum) edit distance / (minimum) edit sequence. You can find the theory of the process here:

https://web.stanford.edu/class/cs124/lec/med.pdf

Let's implement (simplest) Levenstein Distance / Sequence algorithm (for details see https://en.wikipedia.org/wiki/Levenshtein_distance). Let's start from helper classes (I've changed a bit your implementation of them):

  public enum EditOperationKind : byte {     None,    // Nothing to do     Add,     // Add new character     Edit,    // Edit character into character (including char into itself)     Remove,  // Delete existing character   };    public struct EditOperation {     public EditOperation(char valueFrom, char valueTo, EditOperationKind operation) {       ValueFrom = valueFrom;       ValueTo = valueTo;        Operation = valueFrom == valueTo ? EditOperationKind.None : operation;     }      public char ValueFrom { get; }     public char ValueTo {get ;}     public EditOperationKind Operation { get; }      public override string ToString() {       switch (Operation) {         case EditOperationKind.None:           return $"'{ValueTo}' Equal";         case EditOperationKind.Add:           return $"'{ValueTo}' Add";         case EditOperationKind.Remove:           return $"'{ValueFrom}' Remove";         case EditOperationKind.Edit:           return $"'{ValueFrom}' to '{ValueTo}' Edit";         default:           return "???";       }     }   } 

As far as I can see from the examples provided we don't have any edit operation, but add + remove; that's why I've put editCost = 2 when insertCost = 1, int removeCost = 1 (in case of tie: insert + remove vs. edit we put insert + remove). Now we are ready to implement Levenstein algorithm:

public static EditOperation[] EditSequence(   string source, string target,    int insertCost = 1, int removeCost = 1, int editCost = 2) {    if (null == source)     throw new ArgumentNullException("source");   else if (null == target)     throw new ArgumentNullException("target");    // Forward: building score matrix    // Best operation (among insert, update, delete) to perform    EditOperationKind[][] M = Enumerable     .Range(0, source.Length + 1)     .Select(line => new EditOperationKind[target.Length + 1])     .ToArray();    // Minimum cost so far   int[][] D = Enumerable     .Range(0, source.Length + 1)     .Select(line => new int[target.Length + 1])     .ToArray();    // Edge: all removes   for (int i = 1; i <= source.Length; ++i) {     M[i][0] = EditOperationKind.Remove;     D[i][0] = removeCost * i;   }    // Edge: all inserts    for (int i = 1; i <= target.Length; ++i) {     M[0][i] = EditOperationKind.Add;     D[0][i] = insertCost * i;   }    // Having fit N - 1, K - 1 characters let's fit N, K   for (int i = 1; i <= source.Length; ++i)     for (int j = 1; j <= target.Length; ++j) {       // here we choose the operation with the least cost       int insert = D[i][j - 1] + insertCost;       int delete = D[i - 1][j] + removeCost;       int edit = D[i - 1][j - 1] + (source[i - 1] == target[j - 1] ? 0 : editCost);        int min = Math.Min(Math.Min(insert, delete), edit);        if (min == insert)          M[i][j] = EditOperationKind.Add;       else if (min == delete)         M[i][j] = EditOperationKind.Remove;       else if (min == edit)         M[i][j] = EditOperationKind.Edit;        D[i][j] = min;     }    // Backward: knowing scores (D) and actions (M) let's building edit sequence   List<EditOperation> result =      new List<EditOperation>(source.Length + target.Length);    for (int x = target.Length, y = source.Length; (x > 0) || (y > 0);) {     EditOperationKind op = M[y][x];      if (op == EditOperationKind.Add) {       x -= 1;       result.Add(new EditOperation('\0', target[x], op));     }     else if (op == EditOperationKind.Remove) {       y -= 1;       result.Add(new EditOperation(source[y], '\0', op));     }     else if (op == EditOperationKind.Edit) {       x -= 1;       y -= 1;       result.Add(new EditOperation(source[y], target[x], op));     }     else // Start of the matching (EditOperationKind.None)       break;   }    result.Reverse();    return result.ToArray(); } 

Demo:

var sequence = EditSequence("asdfghjk", "wsedrftr");   Console.Write(string.Join(Environment.NewLine, sequence)); 

Outcome:

'a' Remove 'w' Add 's' Equal 'e' Add 'd' Equal 'r' Add 'f' Equal 'g' Remove 'h' Remove 'j' Remove 'k' Remove 't' Add 'r' Add 

Answers 2

I'll go out on a limb here and provide an algorithm that's not the most efficient, but is easy to reason about.

Let's cover some ground first:

1) Order matters

string before = "bar foo" string after = "foo bar" 

Even though "bar" and "foo" occur in both strings, "bar" will need to be removed and added again later. This also tells us it's the after string that gives us the order of chars we're interested in, we want "foo" first.

2) Order over count

Another way to look at it, is that some chars may never get their turn.

string before = "abracadabra" string after = "bar bar" 

Only the bold chars of "bar bar", get their say in "abracadabra". Even though we've got two b's in both strings, only the first one counts. By the time we get to the second b in "bar bar" the second b in "abracadabra" has already been passed, when we were looking for the first occurrence of 'r'.

3) Barriers

Barriers are the chars that exist in both strings, taking order and count into consideration. This already suggests a set might not be the most appropriate data structure, as we would lose count.

For an input

string before = "pinata" string after = "accidental" 

We get (pseudocode)

var barriers = { 'a', 't', 'a' } 

"pinata"

"accidental"

Let's follow the execution flow:

  • 'a' is the first barrier, it's also the first char of after so everything prepending the first 'a' in before can be removed. "pinata" -> "ata"
  • the second barrier is 't', it's not at the next position in our after string, so we can insert everything in between. "ata" -> "accidenta"
  • the third barrier 'a' is already at the next position, so we can move to the next barrier without doing any real work.
  • there are no more barriers, but our string length is still less than that of after, so there will be some post processing. "accidenta" -> "accidental"

Note 'i' and 'n' don't get to play, again, order over count.


Implementation

We've established that order and count matter, a Queue comes to mind.

static public List<Difference> CalculateDifferences(string before, string after) {     List<Difference> result = new List<Difference>();     Queue<char> barriers = new Queue<char>();      #region Preprocessing     int index = 0;     for (int i = 0; i < after.Length; i++)     {         // Look for the first match starting at index         int match = before.IndexOf(after[i], index);         if (match != -1)         {             barriers.Enqueue(after[i]);             index = match + 1;         }     }     #endregion      #region Queue Processing     index = 0;     while (barriers.Any())     {         char barrier = barriers.Dequeue();         // Get the offset to the barrier in both strings,          // ignoring the part that's already been handled         int offsetBefore = before.IndexOf(barrier, index) - index;         int offsetAfter = after.IndexOf(barrier, index) - index;         // Remove prefix from 'before' string         if (offsetBefore > 0)         {             RemoveChars(before.Substring(index, offsetBefore), result);             before = before.Substring(offsetBefore);         }         // Insert prefix from 'after' string         if (offsetAfter > 0)         {             string substring = after.Substring(index, offsetAfter);             AddChars(substring, result);             before = before.Insert(index, substring);             index += substring.Length;         }         // Jump over the barrier         KeepChar(barrier, result);         index++;     }     #endregion      #region Post Queue processing     if (index < before.Length)     {         RemoveChars(before.Substring(index), result);     }     if (index < after.Length)     {         AddChars(after.Substring(index), result);     }     #endregion      return result; }  static private void KeepChar(char barrier, List<Difference> result) {     result.Add(new Difference()     {         c = barrier,         op = Operation.Equal     }); }  static private void AddChars(string substring, List<Difference> result) {     result.AddRange(substring.Select(x => new Difference()     {         c = x,         op = Operation.Add     })); }  static private void RemoveChars(string substring, List<Difference> result) {     result.AddRange(substring.Select(x => new Difference()     {         c = x,         op = Operation.Remove     })); } 

Answers 3

I tested with 3 examples above, and it returns the expected result properly and perfectly.

        int flag = 0;         int flag_2 = 0;          string a = "asdfghjk";         string b = "wsedrftr";          char[] array_a = a.ToCharArray();         char[] array_b = b.ToCharArray();          for (int i = 0,j = 0, n= 0; i < array_b.Count(); i++)         {                //Execute 1 time until reach first equal character                if(i == 0 && a.Contains(array_b[0]))             {                 while (array_a[n] != array_b[0])                 {                     Console.WriteLine(String.Concat(array_a[n], " : Remove"));                     n++;                 }                 Console.WriteLine(String.Concat(array_a[n], " : Equal"));                 n++;             }             else if(i == 0 && !a.Contains(array_b[0]))             {                 Console.WriteLine(String.Concat(array_a[n], " : Remove"));                 n++;                 Console.WriteLine(String.Concat(array_b[0], " : Add"));             }               else             {                 if(n < array_a.Count())                 {                     if (array_a[n] == array_b[i])                     {                         Console.WriteLine(String.Concat(array_a[n], " : Equal"));                         n++;                     }                     else                     {                         flag = 0;                         for (int z = n; z < array_a.Count(); z++)                         {                                                           if (array_a[z] == array_b[i])                             {                                 flag = 1;                                 break;                             }                                                                                       }                          if (flag == 0)                         {                             flag_2 = 0;                             for (int aa = i; aa < array_b.Count(); aa++)                             {                                 for(int bb = n; bb < array_a.Count(); bb++)                                 {                                     if (array_b[aa] == array_a[bb])                                     {                                         flag_2 = 1;                                         break;                                     }                                 }                             }                              if(flag_2 == 1)                             {                                 Console.WriteLine(String.Concat(array_b[i], " : Add"));                             }                             else                             {                                 for (int z = n; z < array_a.Count(); z++)                                 {                                     Console.WriteLine(String.Concat(array_a[z], " : Remove"));                                     n++;                                 }                                  Console.WriteLine(String.Concat(array_b[i], " : Add"));                             }                          }                         else                         {                             Console.WriteLine(String.Concat(array_a[n], " : Remove"));                             i--;                             n++;                         }                      }                 }                 else                 {                     Console.WriteLine(String.Concat(array_b[i], " : Add"));                 }              }          }//end for           MessageBox.Show("Done");       //OUTPUT CONSOLE:     /*     a : Remove     w : Add     s : Equal     e : Add     d : Equal     r : Add     f : Equal     g : Remove     h : Remove     j : Remove     k : Remove     t : Add     r : Add     */   

Answers 4

Here might be another solution, full code and commented. However the result of your first original example is inverted :

class Program {     enum CharState     {         Add,         Equal,         Remove     }      struct CharResult     {         public char c;         public CharState state;     }      static void Main(string[] args)     {         string a = "asdfghjk";         string b = "wsedrftr";         while (true)         {             Console.WriteLine("Enter string a (enter to quit) :");             a = Console.ReadLine();             if (a == string.Empty)                 break;             Console.WriteLine("Enter string b :");             b = Console.ReadLine();              List<CharResult> result = calculate(a, b);             DisplayResults(result);         }         Console.WriteLine("Press a key to exit");         Console.ReadLine();     }      static List<CharResult> calculate(string a, string b)     {         List<CharResult> res = new List<CharResult>();         int i = 0, j = 0;          char[] array_a = a.ToCharArray();         char[] array_b = b.ToCharArray();          while (i < array_a.Length && j < array_b.Length)         {             //For the current char in a, we check for the equal in b             int index = b.IndexOf(array_a[i], j);             if (index < 0) //not found, this char should be removed             {                 res.Add(new CharResult() { c = array_a[i], state = CharState.Remove });                 i++;             }             else             {                 //we add all the chars between B's current index and the index                 while (j < index)                 {                     res.Add(new CharResult() { c = array_b[j], state = CharState.Add });                     j++;                 }                 //then we say the current is the same                 res.Add(new CharResult() { c = array_a[i], state = CharState.Equal });                 i++;                 j++;             }         }          while (i < array_a.Length)         {             //b is now empty, we remove the remains             res.Add(new CharResult() { c = array_a[i], state = CharState.Remove });             i++;         }         while (j < array_b.Length)         {             //a has been treated, we add the remains             res.Add(new CharResult() { c = array_b[j], state = CharState.Add });             j++;         }          return res;     }      static void DisplayResults(List<CharResult> results)     {         foreach (CharResult r in results)         {             Console.WriteLine($"'{r.c}' - {r.state}");         }     } } 

Answers 5

If you want to have a precise comparison between two strings, you must read and understand Levenshtein Distance. by using this algorithm you can precisely calculate rate of similarity between two string and also you can backtrack the algorithm to get the chain of changing on the second string. this algorithm is a important metric for Natural Language Processing also.

there are some other benefits and it's need time to learn.

in this link there is a C# version of Levenshtein Distance :

https://www.dotnetperls.com/levenshtein

Read More

Monday, August 27, 2018

Custom locale configuration for float conversion

Leave a Comment

I need to convert a string in the format "1.234.345,00" to the float value 1234345.00.

One way is to use repeated str.replace:

x = "1.234.345,00" res = float(x.replace('.', '').replace(',', '.'))  print(res, type(res)) 1234345.0 <class 'float'> 

However, this appears manual and non-generalised. This heavily upvoted answer suggests using the locale library. But my default locale doesn't have the same conventions as my input string. I then discovered a way to extract the characters used in local conventions as a dictionary:

import locale  print(locale.localeconv())  {'int_curr_symbol': '', 'currency_symbol': '', 'mon_decimal_point': '',  ..., 'decimal_point': '.', 'thousands_sep': '', 'grouping': []} 

Is there a way to update this dictionary, save as a custom locale and then be able to call this custom locale going forwards. Something like:

mylocale = locale.create_new_locale()  # "blank" conventions or copied from default mylocale.localeconv()['thousands_sep'] = '.' mylocale.localeconv()['decimal_point'] = ','  setlocale(LC_NUMERIC, mylocale) atof('123.456,78')  # 123456.78 

If this isn't possible, how do we get a list of all available locale and their conventions? Seems anti-pattern to "deduce" the correct configuration from the conventions (not to mention inefficient / manual), so I was hoping for a generic solution such as above pseudo-code.


Edit: Here's my attempt at finding all locales where thousands_sep == '.' and decimal_point == ','. In fact, more generally, to group locales by combinations of these parameters:

import locale from collections import defaultdict  d = defaultdict(list)  for alias in locale.locale_alias:     locale.setlocale(locale.LC_ALL, alias)     env = locale.localeconv()     d[(env['thousands_sep'], env['decimal_point'])].append(alias) 

Result:

--------------------------------------------------------------------------- Error                                     Traceback (most recent call last) <ipython-input-164-f8f6a6db7637> in <module>()       5        6 for alias in locale.locale_alias: ----> 7     locale.setlocale(locale.LC_ALL, alias)       8     env = locale.localeconv()       9     d[(env['thousands_sep'], env['decimal_point'])].append(alias)  C:\Program Files\Anaconda3\lib\locale.py in setlocale(category, locale)     596         # convert to string     597         locale = normalize(_build_localename(locale)) --> 598     return _setlocale(category, locale)     599      600 def resetlocale(category=LC_ALL):  Error: unsupported locale setting 

3 Answers

Answers 1

If you pop open the source code for locale, you can see that there is a variable called _override_localeconv (which seems to be for testing purposes).

# With this dict, you can override some items of localeconv's return value. # This is useful for testing purposes. _override_localeconv = {} 

Trying the following does seem to override the dictionary without changing the entire locale, though it probably has some unintended consequences, especially since changing locales isn't threadsafe. Be careful!

import locale  locale._override_localeconv["thousands_sep"] = "." locale._override_localeconv["decimal_point"] = ","  print locale.atof('123.456,78') 

Try it online!

Answers 2

Here's something, using Babel, that works for me.

First you feed it some test data, with your expectations and it builds a dictionary of separator to locale alias that fits.

Then you can convert from that point on.

import string from decimal import Decimal from babel.numbers import parse_decimal, NumberFormatError from babel.core import UnknownLocaleError import locale  traindata = [     ("1.234.345,00", Decimal("1234345.00")),     ("1,234,345.00", Decimal("1234345.00")),     ("345", Decimal("345.00")), ]  data = traindata + [     ("345,00", Decimal("345.00")),     ("345.00", Decimal("345.00")),     ("746", Decimal("746.00")), ]  def findseps(input_):     #you need to have no separator      #or at least a decimal separator for this to work...      seps = [c for c in input_ if not c in string.digits]     if not seps:         return ""      sep = seps[-1]     #if the decimal is something then thousand will be the other...     seps = "." + sep if sep == "," else "," + sep     return seps    def setup(input_, exp, lookup):       key = findseps(input_)      if key in lookup:         return      for alias in locale.locale_alias:         #print(alias)          try:             got = parse_decimal(input_, locale=alias)         except (NumberFormatError,UnknownLocaleError, ValueError) as e:             continue         except (Exception,) as e:             raise         if exp == got:             lookup[key] = alias             return   def convert(input_, lookup):     seps = findseps(input_)     try:         locale_ = lookup[seps]         convert.locale_ = locale_     except (KeyError,) as e:         convert.locale_ = None         return "unexpected seps:%s" % seps      try:         return parse_decimal(input_, locale=locale_)     except (Exception,) as e:         return e   lookup = {}  #train your data for input_, exp in traindata:     setup(input_, exp, lookup)  #once it's trained you know which locales to use print(data)   for input_, exp in data:     got = convert(input_, lookup)      # print (input_)     msg = "%s => %s with local:%s:" % (input_, got, convert.locale_)     if exp == got:         print("\n  success : " + msg)     else:         print("\n  failure : " + msg)  print(lookup) 

output:

[('1.234.345,00', Decimal('1234345.00')), ('1,234,345.00', Decimal('1234345.00')), ('345', Decimal('345.00')), ('345,00', Decimal('345.00')), ('345.00', Decimal('345.00')), ('746', Decimal('746.00'))]    success : 1.234.345,00 => 1234345.00 with local:is_is:    success : 1,234,345.00 => 1234345.00 with local:ko_kr.euc:    success : 345 => 345 with local:ko_kr.euc:    success : 345,00 => 345.00 with local:is_is:    success : 345.00 => 345.00 with local:ko_kr.euc:    success : 746 => 746 with local:ko_kr.euc: {',.': 'ko_kr.euc', '': 'ko_kr.euc', '.,': 'is_is'} 

Answers 3

There are two parts in your question:

  1. How can I parse '1.234.345,00' in a generic way?
  2. How can I easily find the locale associated to '1.234.345,00'?

You can use the amazing Babel library for both.

How can I parse '1.234.345,00' in a generic way?

One locale associated with a . thousands separator and a , decimal separator is ger_de, for German.

To parse it, simply use

>>> from babel.numbers import parse_decimal >>> parse_decimal('1.234.345,00', locale='ger_de') Decimal('1234345.00') 

How can I easily find the locale associated to '1.234.345,00'?

Use this routine which checks the string to parse against the expected value for all locales, and returns the ones that are compatible:

import locale from babel.numbers import parse_decimal from decimal import Decimal  def get_compatible_locales(string_to_parse, expected_decimal):     compatible_aliases = []     for alias in locale.locale_alias:         try:             parsed_decimal = parse_decimal(string_to_parse, locale=alias)             if parsed_decimal == expected_decimal:                 compatible_aliases.append(alias)         except Exception:             continue     return compatible_aliases 

For your example:

>>> print(get_compatible_locales('1.234.345,00', Decimal('1234345'))) ['ar_dz', 'ar_lb', 'ar_ly', 'ar_ma', 'ar_tn', 'ast_es', 'az', 'az_az', 'az_az.iso88599e', 'bs', 'bs_ba', 'ca', 'ca_ad', 'ca_es', 'ca_es@valencia', 'ca_fr', 'ca_it', 'da', 'da_dk', 'de', 'de_at', 'de_be', 'de_de', 'de_lu', 'el', 'el_cy', 'el_gr', 'el_gr@euro', 'en_be', 'en_dk', 'es', 'es_ar', 'es_bo', 'es_cl', 'es_co', 'es_ec', 'es_es', 'es_py', 'es_uy', 'es_ve', 'eu', 'eu_es', 'fo', 'fo_fo', 'fr_lu', 'fy_nl', 'ger_de', 'gl', 'gl_es', 'hr', 'hr_hr', 'hsb_de', 'id', 'id_id', 'in', 'in_id', 'is', 'is_is', 'it', 'it_it', 'kl', 'kl_gl', 'km_kh', 'lb_lu', 'lo', 'lo_la', 'lo_la.cp1133', 'lo_la.ibmcp1133', 'lo_la.mulelao1', 'mk', 'mk_mk', 'nl', 'nl_aw', 'nl_be', 'nl_nl', 'ps_af', 'pt', 'pt_br', 'ro', 'ro_ro', 'rw', 'rw_rw', 'sl', 'sl_si', 'sr', 'sr@cyrillic', 'sr@latn', 'sr_cs', 'sr_cs.iso88592@latn', 'sr_cs@latn', 'sr_me', 'sr_rs', 'sr_rs@latn', 'sr_yu', 'sr_yu.cp1251@cyrillic', 'sr_yu.iso88592', 'sr_yu.iso88595', 'sr_yu.iso88595@cyrillic', 'sr_yu.microsoftcp1251@cyrillic', 'sr_yu.utf8', 'sr_yu.utf8@cyrillic', 'sr_yu@cyrillic', 'tr', 'tr_cy', 'tr_tr', 'vi', 'vi_vn', 'vi_vn.tcvn', 'vi_vn.tcvn5712', 'vi_vn.viscii', 'vi_vn.viscii111', 'wo_sn'] 

Bonus: How can I have a human-readable version of these locales?

Use the following routine, where my_locale should be your own locale:

from babel import Locale  def get_display_name(alias, my_locale='en_US'):     l = Locale.parse(alias)     return l.get_display_name(my_locale) 

You can then use it this way:

>>> print({loc: get_display_name(loc) for loc in locales}) {'ar_dz': 'Arabic (Algeria)', 'ar_lb': 'Arabic (Lebanon)', 'ar_ly': 'Arabic (Libya)', 'ar_ma': 'Arabic (Morocco)', 'ar_tn': 'Arabic (Tunisia)', 'ast_es': 'Asturian (Spain)', 'az': 'Azerbaijani', 'az_az': 'Azerbaijani (Latin, Azerbaijan)', 'az_az.iso88599e': 'Azerbaijani (Latin, Azerbaijan)', 'bs': 'Bosnian', 'bs_ba': 'Bosnian (Latin, Bosnia & Herzegovina)', 'ca': 'Catalan', 'ca_ad': 'Catalan (Andorra)', 'ca_es': 'Catalan (Spain)', 'ca_es@valencia': 'Catalan (Spain)', 'ca_fr': 'Catalan (France)', 'ca_it': 'Catalan (Italy)', 'da': 'Danish', 'da_dk': 'Danish (Denmark)', 'de': 'German', 'de_at': 'German (Austria)', 'de_be': 'German (Belgium)', 'de_de': 'German (Germany)', 'de_lu': 'German (Luxembourg)', 'el': 'Greek', 'el_cy': 'Greek (Cyprus)', 'el_gr': 'Greek (Greece)', 'el_gr@euro': 'Greek (Greece)', 'en_be': 'English (Belgium)', 'en_dk': 'English (Denmark)', 'es': 'Spanish', 'es_ar': 'Spanish (Argentina)', 'es_bo': 'Spanish (Bolivia)', 'es_cl': 'Spanish (Chile)', 'es_co': 'Spanish (Colombia)', 'es_ec': 'Spanish (Ecuador)', 'es_es': 'Spanish (Spain)', 'es_py': 'Spanish (Paraguay)', 'es_uy': 'Spanish (Uruguay)', 'es_ve': 'Spanish (Venezuela)', 'eu': 'Basque', 'eu_es': 'Basque (Spain)', 'fo': 'Faroese', 'fo_fo': 'Faroese (Faroe Islands)', 'fr_lu': 'French (Luxembourg)', 'fy_nl': 'Western Frisian (Netherlands)', 'ger_de': 'German (Germany)', 'gl': 'Galician', 'gl_es': 'Galician (Spain)', 'hr': 'Croatian', 'hr_hr': 'Croatian (Croatia)', 'hsb_de': 'Upper Sorbian (Germany)', 'id': 'Indonesian', 'id_id': 'Indonesian (Indonesia)', 'in': 'Indonesian (Indonesia)', 'in_id': 'Indonesian (Indonesia)', 'is': 'Icelandic', 'is_is': 'Icelandic (Iceland)', 'it': 'Italian', 'it_it': 'Italian (Italy)', 'kl': 'Kalaallisut', 'kl_gl': 'Kalaallisut (Greenland)', 'km_kh': 'Khmer (Cambodia)', 'lb_lu': 'Luxembourgish (Luxembourg)', 'lo': 'Lao', 'lo_la': 'Lao (Laos)', 'lo_la.cp1133': 'Lao (Laos)', 'lo_la.ibmcp1133': 'Lao (Laos)', 'lo_la.mulelao1': 'Lao (Laos)', 'mk': 'Macedonian', 'mk_mk': 'Macedonian (Macedonia)', 'nl': 'Dutch', 'nl_aw': 'Dutch (Aruba)', 'nl_be': 'Dutch (Belgium)', 'nl_nl': 'Dutch (Netherlands)', 'ps_af': 'Pashto (Afghanistan)', 'pt': 'Portuguese', 'pt_br': 'Portuguese (Brazil)', 'ro': 'Romanian', 'ro_ro': 'Romanian (Romania)', 'rw': 'Kinyarwanda', 'rw_rw': 'Kinyarwanda (Rwanda)', 'sl': 'Slovenian', 'sl_si': 'Slovenian (Slovenia)', 'sr': 'Serbian', 'sr@cyrillic': 'Serbian', 'sr@latn': 'Serbian', 'sr_cs': 'Serbian (Cyrillic, Serbia)', 'sr_cs.iso88592@latn': 'Serbian (Cyrillic, Serbia)', 'sr_cs@latn': 'Serbian (Cyrillic, Serbia)', 'sr_me': 'Serbian (Latin, Montenegro)', 'sr_rs': 'Serbian (Cyrillic, Serbia)', 'sr_rs@latn': 'Serbian (Cyrillic, Serbia)', 'sr_yu': 'Serbian (Cyrillic, Serbia)', 'sr_yu.cp1251@cyrillic': 'Serbian (Cyrillic, Serbia)', 'sr_yu.iso88592': 'Serbian (Cyrillic, Serbia)', 'sr_yu.iso88595': 'Serbian (Cyrillic, Serbia)', 'sr_yu.iso88595@cyrillic': 'Serbian (Cyrillic, Serbia)', 'sr_yu.microsoftcp1251@cyrillic': 'Serbian (Cyrillic, Serbia)', 'sr_yu.utf8': 'Serbian (Cyrillic, Serbia)', 'sr_yu.utf8@cyrillic': 'Serbian (Cyrillic, Serbia)', 'sr_yu@cyrillic': 'Serbian (Cyrillic, Serbia)', 'tr': 'Turkish', 'tr_cy': 'Turkish (Cyprus)', 'tr_tr': 'Turkish (Turkey)', 'vi': 'Vietnamese', 'vi_vn': 'Vietnamese (Vietnam)', 'vi_vn.tcvn': 'Vietnamese (Vietnam)', 'vi_vn.tcvn5712': 'Vietnamese (Vietnam)', 'vi_vn.viscii': 'Vietnamese (Vietnam)', 'vi_vn.viscii111': 'Vietnamese (Vietnam)', 'wo_sn': 'Wolof (Senegal)'} 

Try it online!

Read More

Sunday, April 22, 2018

Check if a Javascript string is a url

Leave a Comment

Is there a way in javascript to check if a string is a url?

RegExes are excluded because the url is most likely written like stackoverflow; that is to say that it might not have a .com, www or http

16 Answers

Answers 1

A related question with an answer:

Javascript regex URL matching

Or this Regexp from Devshed:

function ValidURL(str) {   var pattern = new RegExp('^(https?:\/\/)?'+ // protocol     '((([a-z\d]([a-z\d-]*[a-z\d])*)\.)+[a-z]{2,}|'+ // domain name     '((\d{1,3}\.){3}\d{1,3}))'+ // OR ip (v4) address     '(\:\d+)?(\/[-a-z\d%_.~+]*)*'+ // port and path     '(\?[;&a-z\d%_.~+=-]*)?'+ // query string     '(\#[-a-z\d_]*)?$','i'); // fragment locater   if(!pattern.test(str)) {     alert("Please enter a valid URL.");     return false;   } else {     return true;   } } 

Answers 2

function isURL(str) {   var pattern = new RegExp('^(https?:\\/\\/)?'+ // protocol   '((([a-z\\d]([a-z\\d-]*[a-z\\d])*)\\.?)+[a-z]{2,}|'+ // domain name   '((\\d{1,3}\\.){3}\\d{1,3}))'+ // OR ip (v4) address   '(\\:\\d+)?(\\/[-a-z\\d%_.~+]*)*'+ // port and path   '(\\?[;&a-z\\d%_.~+=-]*)?'+ // query string   '(\\#[-a-z\\d_]*)?$','i'); // fragment locator   return pattern.test(str); } 

Answers 3

Rather than using a regular expression, I would recommend making use of an anchor element.

when you set the href property of an anchor, various other properties are set.

var parser = document.createElement('a'); parser.href = "http://example.com:3000/pathname/?search=test#hash";  parser.protocol; // => "http:" parser.hostname; // => "example.com" parser.port;     // => "3000" parser.pathname; // => "/pathname/" parser.search;   // => "?search=test" parser.hash;     // => "#hash" parser.host;     // => "example.com:3000" 

source

However, if the value href is bound to is not a valid url, then the value of those auxiliary properties will be the empty string.

Edit: as pointed out in the comments: if an invalid url is used, the properties of the current URL may be substituted.

So, as long as you're not passing in the URL of the current page, you can do something like:

function isValidURL(str) {    var a  = document.createElement('a');    a.href = str;    return (a.host && a.host != window.location.host); } 

Answers 4

To Validate Url using javascript is shown below

function ValidURL(str) {   var regex = /(http|https):\/\/(\w+:{0,1}\w*)?(\S+)(:[0-9]+)?(\/|\/([\w#!:.?+=&%!\-\/]))?/;   if(!regex .test(str)) {     alert("Please enter valid URL.");     return false;   } else {     return true;   } } 

Answers 5

Improvement on the accepted answer...

  • Has double escaping for backslashes (\\)
  • Ensures that domains have a dot and an extension (.com .io .xyz)
  • Allows full colon (:) in the path e.g. http://thingiverse.com/download:1894343
  • Allows ampersand (&) in path e.g http://en.wikipedia.org/wiki/Procter_&_Gamble
  • Allows @ symbol in path e.g. https://medium.com/@techytimo

    isURL(str) {   var pattern = new RegExp('^(https?:\\/\\/)?'+ // protocol   '((([a-z\\d]([a-z\\d-]*[a-z\\d])*)\\.)+[a-z]{2,}|'+ // domain name and extension   '((\\d{1,3}\\.){3}\\d{1,3}))'+ // OR ip (v4) address   '(\\:\\d+)?'+ // port   '(\\/[-a-z\\d%@_.~+&:]*)*'+ // path   '(\\?[;&a-z\\d%@_.,~+&:=-]*)?'+ // query string   '(\\#[-a-z\\d_]*)?$','i'); // fragment locator   return pattern.test(str); } 

Answers 6

Rely on a library: https://www.npmjs.com/package/valid-url

import { isWebUri } from 'valid-url'; // ... if (!isWebUri(url)) {     return "Not a valid url."; } 

Answers 7

You can try to use URL constructor: if it doesn't throw, the string is a valid URL:

const isValidUrl = (string) => {   try {     new URL(string);     return true;   } catch (_) {     return false;     } } 

Answers 8

I can't comment on the post that is the closest #5717133, but below is the way I figured out how to get @tom-gullen regex working.

/^(https?:\/\/)?((([a-z\d]([a-z\d-]*[a-z\d])*)\.)+[a-z]{2,}|((\d{1,3}\.){3}\d{1,3}))(\:\d+)?(\/[-a-z\d%_.~+]*)*(\?[;&a-z\d%_.~+=-]*)?(\#[-a-z\d_]*)?$/i 

Answers 9

(I don't have reps to comment on ValidURL example; hence post this as an answer.)

While use of protocol relative URLs is not encouraged (The Protocol-relative URL), they do get employed sometimes. To validate such an URL with a regular expression the protocol part could be optional, e.g.:

function isValidURL(str) {     var pattern = new RegExp('^((https?:)?\\/\\/)?'+ // protocol         '((([a-z\\d]([a-z\\d-]*[a-z\\d])*)\\.)+[a-z]{2,}|'+ // domain name         '((\\d{1,3}\\.){3}\\d{1,3}))'+ // OR ip (v4) address         '(\\:\\d+)?(\\/[-a-z\\d%_.~+]*)*'+ // port and path         '(\\?[;&a-z\\d%_.~+=-]*)?'+ // query string         '(\\#[-a-z\\d_]*)?$','i'); // fragment locater     if (!pattern.test(str)) {         return false;     } else {         return true;     } } 

As others noted, regular expression does not seem to be the best suited approach for validating URLs, though.

Answers 10

One function that I have been using to validate a URL "string" is:

var matcher = /^(?:\w+:)?\/\/([^\s\.]+\.\S{2}|localhost[\:?\d]*)\S*$/;  function isUrl(string){   return matcher.test(string); } 

This function will return a boolean whether the string is a URL.

Answers 11

As has been noted the perfect regex is elusive but still seems to be a reasonable approach (alternatives are server side tests or the new experimental URL API). However the high ranking answers are often returning false for common URLs but even worse will freeze your app/page for minutes on even as simple a string as isURL('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'). It's been pointed out in some of the comments, but most probably haven't entered a bad value to see it. Hanging like that makes that code unusable in any serious application. I think it's due to the repeated case insensitive sets in code like ((([a-z\\d]([a-z\\d-]*[a-z\\d])*)\\.?)+[a-z]{2,}|' .... Take out the 'i' and it doesn't hang but will of course not work as desired. But even with the ignore case flag those tests reject high unicode values that are allowed.

The best already mentioned is:

function isURL(str) {   return /^(?:\w+:)?\/\/([^\s\.]+\.\S{2}|localhost[\:?\d]*)\S*$/.test(str);  } 

That comes from Github segmentio/is-url. The good thing about a code repository is you can see the testing and any issues and also the test strings run through it. There's a branch that would allow strings missing protocol like google.com.

There is one other repository I've seen that is even better for isURL at dperini/regex-weburl.js, but it is highly complex. It has a bigger test list of valid and invalid URLs. The simple one above still passes all the positives and only fails to block a few odd negatives like http://a.b--c.de/ as well as the special ips.

Whichever you choose, run it through this function which I've adapted from the tests on dperini/regex-weburl.js, while using your browser's Developer Tools inpector.

function testIsURL() { //should match console.assert(isURL("http://foo.com/blah_blah")); console.assert(isURL("http://foo.com/blah_blah/")); console.assert(isURL("http://foo.com/blah_blah_(wikipedia)")); console.assert(isURL("http://foo.com/blah_blah_(wikipedia)_(again)")); console.assert(isURL("http://www.example.com/wpstyle/?p=364")); console.assert(isURL("https://www.example.com/foo/?bar=baz&inga=42&quux")); console.assert(isURL("http://✪df.ws/123")); console.assert(isURL("http://userid:password@example.com:8080")); console.assert(isURL("http://userid:password@example.com:8080/")); console.assert(isURL("http://userid@example.com")); console.assert(isURL("http://userid@example.com/")); console.assert(isURL("http://userid@example.com:8080")); console.assert(isURL("http://userid@example.com:8080/")); console.assert(isURL("http://userid:password@example.com")); console.assert(isURL("http://userid:password@example.com/")); console.assert(isURL("http://142.42.1.1/")); console.assert(isURL("http://142.42.1.1:8080/")); console.assert(isURL("http://➡.ws/䨹")); console.assert(isURL("http://⌘.ws")); console.assert(isURL("http://⌘.ws/")); console.assert(isURL("http://foo.com/blah_(wikipedia)#cite-1")); console.assert(isURL("http://foo.com/blah_(wikipedia)_blah#cite-1")); console.assert(isURL("http://foo.com/unicode_(✪)_in_parens")); console.assert(isURL("http://foo.com/(something)?after=parens")); console.assert(isURL("http://☺.damowmow.com/")); console.assert(isURL("http://code.google.com/events/#&product=browser")); console.assert(isURL("http://j.mp")); console.assert(isURL("ftp://foo.bar/baz")); console.assert(isURL("http://foo.bar/?q=Test%20URL-encoded%20stuff")); console.assert(isURL("http://مثال.إختبار")); console.assert(isURL("http://例子.测试")); console.assert(isURL("http://उदाहरण.परीक्षा")); console.assert(isURL("http://-.~_!$&'()*+,;=:%40:80%2f::::::@example.com")); console.assert(isURL("http://1337.net")); console.assert(isURL("http://a.b-c.de")); console.assert(isURL("http://223.255.255.254")); console.assert(isURL("postgres://u:p@example.com:5702/db")); console.assert(isURL("https://d1f4470da51b49289906b3d6cbd65074@app.getsentry.com/13176"));  //SHOULD NOT MATCH: console.assert(!isURL("http://")); console.assert(!isURL("http://.")); console.assert(!isURL("http://..")); console.assert(!isURL("http://../")); console.assert(!isURL("http://?")); console.assert(!isURL("http://??")); console.assert(!isURL("http://??/")); console.assert(!isURL("http://#")); console.assert(!isURL("http://##")); console.assert(!isURL("http://##/")); console.assert(!isURL("http://foo.bar?q=Spaces should be encoded")); console.assert(!isURL("//")); console.assert(!isURL("//a")); console.assert(!isURL("///a")); console.assert(!isURL("///")); console.assert(!isURL("http:///a")); console.assert(!isURL("foo.com")); console.assert(!isURL("rdar://1234")); console.assert(!isURL("h://test")); console.assert(!isURL("http:// shouldfail.com")); console.assert(!isURL(":// should fail")); console.assert(!isURL("http://foo.bar/foo(bar)baz quux")); console.assert(!isURL("ftps://foo.bar/")); console.assert(!isURL("http://-error-.invalid/")); console.assert(!isURL("http://a.b--c.de/")); console.assert(!isURL("http://-a.b.co")); console.assert(!isURL("http://a.b-.co")); console.assert(!isURL("http://0.0.0.0")); console.assert(!isURL("http://10.1.1.0")); console.assert(!isURL("http://10.1.1.255")); console.assert(!isURL("http://224.1.1.1")); console.assert(!isURL("http://1.1.1.1.1")); console.assert(!isURL("http://123.123.123")); console.assert(!isURL("http://3628126748")); console.assert(!isURL("http://.www.foo.bar/")); console.assert(!isURL("http://www.foo.bar./")); console.assert(!isURL("http://.www.foo.bar./")); console.assert(!isURL("http://10.1.1.1"));} 

And then test that string of 'a's.

Answers 12

I am using below function to validate URL with or without http/https:

function isValidURL(string) {    var res = string.match(/(http(s)?:\/\/.)?(www\.)?[-a-zA-Z0-9@:%._\+~#=]{2,256}\.[a-z]{2,6}\b([-a-zA-Z0-9@:%_\+.~#?&//=]*)/g);    if (res == null)      return false;    else      return true;  };    var testCase1 = "http://en.wikipedia.org/wiki/Procter_&_Gamble";  console.log(isValidURL(testCase1)); // return true    var testCase2 = "http://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&docid=nIv5rk2GyP3hXM&tbnid=isiOkMe3nCtexM:&ved=0CAUQjRw&url=http%3A%2F%2Fanimalcrossing.wikia.com%2Fwiki%2FLion&ei=ygZXU_2fGKbMsQTf4YLgAQ&bvm=bv.65177938,d.aWc&psig=AFQjCNEpBfKnal9kU7Zu4n7RnEt2nerN4g&ust=1398298682009707";  console.log(isValidURL(testCase2)); // return true    var testCase3 = "https://sdfasd";  console.log(isValidURL(testCase3)); // return false    var testCase4 = "dfdsfdsfdfdsfsdfs";  console.log(isValidURL(testCase4)); // return false    var testCase5 = "magnet:?xt=urn:btih:123";  console.log(isValidURL(testCase5)); // return false    var testCase6 = "https://stackoverflow.com/";  console.log(isValidURL(testCase6)); // return true    var testCase7 = "https://w";  console.log(isValidURL(testCase7)); // return false    var testCase8 = "https://sdfasdp.ppppppppppp";  console.log(isValidURL(testCase8)); // return false

Answers 13

You can use the URL native API:

  const isUrl = string => {       try { return Boolean(new URL(string)); }       catch(e){ return false; }   } 

Answers 14

Here is yet another method.

var elm;  function isValidURL(u){    if(!elm){      elm = document.createElement('input');      elm.setAttribute('type', 'url');    }    elm.value = u;    return elm.validity.valid;  }    console.log(isValidURL('http://www.google.com/'));  console.log(isValidURL('//google.com'));  console.log(isValidURL('google.com'));  console.log(isValidURL('localhost:8000'));

Answers 15

The question asks a validation method for an url such as stackoverflow, without the protocol or any dot in the hostname. So, it's not a matter of validating url sintax, but checking if it's a valid url, by actually calling it.

I tried several methods for knowing if the url true exists and is callable from within the browser, but did not find any way to test with javascript the response header of the call:

  • adding an anchor element is fine for firing the click() method.
  • making ajax call to the challenging url with 'GET' is fine, but has it's various limitations due to CORS policies and it is not the case of using ajax, for as the url maybe any outside my server's domain.
  • using the fetch API has a workaround similar to ajax.
  • other problem is that I have my server under https protocol and throws an exception when calling non secure urls.

So, the best solution I can think of is getting some tool to perform CURL using javascript trying something like curl -I <url>. Unfortunately I did not find any and in appereance it's not possible. I will appreciate any comments on this.

But, in the end, I have a server running PHP and as I use Ajax for almost all my requests, I wrote a function on the server side to perform the curl request there and return to the browser.

Regarding the single word url on the question 'stackoverflow' it will lead me to https://daniserver.com.ar/stackoverflow, where daniserver.com.ar is my own domain.

Answers 16

I think using the native URL API is better than a complex regex patterns as @pavlo suggested. It has some drawbacks though which we can fix by some extra code. This approach fails for the following valid url.

//cdn.google.com/script.js 

We can add the missing protocol beforehand to avoid that. It also fails to detect following invalid url.

http://w http://.. 

So why check the whole url? we can just check the domain. I borrowed the regex to verify domain from here.

function isValidUrl(string) {     if (string && string.length > 1 && string.slice(0, 2) == '//') {         string = 'http:' + string; //dummy protocol so that URL works     }     try {         var url = new URL(string);         return url.hostname && url.hostname.match(/^([a-z0-9])(([a-z0-9-]{1,61})?[a-z0-9]{1})?(\.[a-z0-9](([a-z0-9-]{1,61})?[a-z0-9]{1})?)?(\.[a-zA-Z]{2,4})+$/) ? true : false;     } catch (_) {         return false;     } } 

The hostname attribute is empty string for javascript:void(0), so it works for that too, and you can also add IP address verifier too. I'd like to stick to native API's most, and hope it starts to support everything in near future.

Read More

Thursday, April 12, 2018

Hamburger menu icon ☰ not appearing on Japanese safari

Leave a Comment

I have some basic HTML that contains a hamburger menu character, something like this:

<div>☰</div> 

The problem is that the character is not visible on Japanese browsers. I have <meta charset="utf-8"> in the head.

What would I need to do to make it work?

Thanks.

7 Answers

Answers 1

Maybe you can get better results using svg.

You must check if the font you are using contains that character.

<svg encoding="UTF-8" >    <text x="10"  y="50" font-size="55">      &#8801;    </text>  </svg>

You can also set the font family from here. (Import?)

See https://developer.mozilla.org/fr/docs/Web/SVG/Element/text

There is alternative icons that can maybe works directly. (Source from personal script)

☰TRIGRAM FOR HEAVEN Hex: 2630 | Dec: 9776 ☱TRIGRAM FOR LAKE Hex: 2631 | Dec: 9777 ☲TRIGRAM FOR FIRE Hex: 2632 | Dec: 9778 ☳TRIGRAM FOR THUNDER Hex: 2633 | Dec: 9779 ☴TRIGRAM FOR WIND Hex: 2634 | Dec: 9780 ☵TRIGRAM FOR WATER Hex: 2635 | Dec: 9781 ☶TRIGRAM FOR MOUNTAIN Hex: 2636 | Dec: 9782 ☷TRIGRAM FOR EARTH Hex: 1D0D3 | Dec: 118995 

Check also the following snippet: It will generate all unicodes chars that your browser can currently display. It may ease your search!

var i = 0      do document.write("<a title='(Linux|Hex): [CTRL+SHIFT]+u"+(i).toString(16)+"\nHtml entity: &# "+i+";\n&#x"+(i).toString(16)+";\n(Win|Dec): [ALT]+"+i+"' onmouseover='this.focus()' onclick='this.href=\"//google.com/?q=\"+this.innerHTML' style='cursor:pointer' target='new'>"+"&#"+i+";</a>"),i++      while (i<136690)  window.stop() 
From What characters can be used for up/down triangle (arrow without stem) for display in HTML?

Answers 2

Without access to a Japanese browser I'm assuming (as others have) that the issue is at the font level. i.e. there is no glyph for the code point U+2630.

Assuming that your page is also served correctly with the encoding as Content-Type: text/html; charset=UTF-8 the issue of HTML encoding should be redundant.

Personally I would use an icon font like Font Awesome to guarantee rendering in all browsers/platforms. Japanese browsers aside, you just don't know what platforms have missing glyphs in their fonts.

As others have pointed out, font files can be large, so I would use the excellent IcoMoon to create a custom font with only the glyphs I need. I do this on my site with 94 icons from various packages and the font files are about 23k.

Alternatively (and for this example, quickest) you could use a standard Unicode font like Google's Noto Sans. Google fonts also allow you to request only the symbols you need, so you can keep the size small there too.

See my example Codepen pulling just one glyph from Noto Sans.

Answers 3

There may be reason that there is no font in the system where the browser runs contains a glyph for “☰” U+2630 TRIGRAM FOR HEAVEN.

Below are some other options to achieve this:

  1. Create your own Menu icon using pure CSS and HTML like below:

.hamburger-icon {    padding: 19px 16px;    display: inline-block;    position: absolute;    top: 0;    left: 0;  }    .hamburger-icon span {    width: 40px;    background-color: #000;    height: 5px;    display: block;    margin-bottom: 6px;  }    .hamburger-icon span:last-child {    margin-bottom: 0px;  }
<div>    <label class="hamburger-icon">      <span>&nbsp;</span>      <span>&nbsp;</span>      <span>&nbsp;</span>   </label>  </div>

  1. Use an image of Menu icon

  2. Use a downloadable font with @font-face. This will take few megabytes to load.

Answers 4

Try this:

<div>&#8801;</div>

By using the character UNICODE U+2261 (8801), or called by IDENTICAL TO, seems it will reduce font support issues on devices. Or use

<div>&equiv;</div>

According to my experience, it works most of the time.

Also, this link could help you. http://graphemica.com/%E2%89%A1

Answers 5

Why you don't just take a screen of that symbol?

Like <div> <img src="example.png"> </div> ?

Answers 6

Simply:

<div>&#9776;</div>

Answers 7

You could always use bootstrap for your hamburger menu. Here is a link to the tutorial - https://mdbootstrap.com/components/bootstrap-hamburger-menu/

Read More

Thursday, March 1, 2018

How to find and replace a particular character but only if it is in quotes?

Leave a Comment

Problem: I have thousands of documents which contains a specific character I don't want. E.g. the character a. These documents contain a variety of characters, but the a's I want to replace are inside double quotes or single quotes.

I would like to find and replace them, and I thought using Regex would be needed. I am using VSCode, but I'm open to any suggestions.

My attempt: I was able to find the following regex to match for a specific string containing the values inside the ().

".*?(r).*?" 

However, this only highlights the entire quote. I want to highlight the character only.

Any solution, perhaps outside of regex, is welcome.

Example outcomes: Given, the character is a, find replace to b

Somebody once told me "apples" are good for you => Somebody once told me "bpples" are good for you

"Aardvarks" make good kebabs => "Abrdvbrks" make good kebabs

The boy said "aaah!" when his mom told him he was eating aardvark => The boy said "bbbh!" when his mom told him he was eating aardvark

6 Answers

Answers 1

Visual Studio Code

VS Code uses JavaScript RegEx engine for its find / replace functionality. This means you are very limited in working with regex in comparison to other flavors like .NET or PCRE.

Lucky enough that this flavor supports lookaheads and with lookaheads you are able to look for but not consume character. So one way to ensure that we are within a quoted string is to look for number of quotes down to bottom of file / subject string to be odd after matching an a:

a(?=[^"]*"[^"]*(?:"[^"]*"[^"]*)*$) 

Live demo

This looks for as in a double quoted string, to have it for single quoted strings substitute all "s with '. You can't have both at a time.

There is a problem with regex above however, that it conflicts with escaped double quotes within double quoted strings. To match them too if it matters you have a long way to go:

a(?=[^"\\]*(?:\\.[^"\\]*)*"[^"\\]*(?:\\.[^"\\]*)*(?:"[^"\\]*(?:\\.[^"\\]*)*"[^"\\]*(?:\\.[^"\\]*)*)*$) 

Applying these approaches on large files probably will result in an stack overflow so let's see a better approach.

I am using VSCode, but I'm open to any suggestions.

That's great. Then I'd suggest to use awk or sed or something more programmatic in order to achieve what you are after or if you are able to use Sublime Text a chance exists to work around this problem in a more elegant way.

Sublime Text

This is supposed to work on large files with hundred of thousands of lines but care that it works for a single character (here a) that with some modifications may work for a word or substring too:

Search for:

(?:"|\G(?<!")(?!\A))(?<r>[^a"\\]*+(?>\\.[^a"\\]*)*+)\K(a|"(*SKIP)(*F))(?(?=((?&r)"))\3)                            ^              ^            ^ 

Replace it with: WHATEVER\3

Live demo

RegEx Breakdown:

(?: # Beginning of non-capturing group #1     "   # Match a `"`     |   # Or     \G(?<!")(?!\A)  # Continue matching from last successful match                     # It shouldn't start right after a `"` )   # End of NCG #1 (?<r>   # Start of capturing group `r`     [^a"\\]*+   # Match anything except `a`, `"` or a backslash (possessively)     (?>\\.[^a"\\]*)*+   # Match an escaped character or                          # repeat last pattern as much as possible )\K     # End of CG `r`, reset all consumed characters (   # Start of CG #2      a   # Match literal `a`     |   # Or     "(*SKIP)(*F)    # Match a `"` and skip over current match ) (?(?=   # Start a conditional cluster, assuming a positive lookahead     ((?&r)")    # Start of CG #3, recurs CG `r` and match `"`   )     # End of condition   \3    # If conditional passed match CG #3  )  # End of conditional 

enter image description here

Three-step approach

Last but not least...

Matching a character inside quotation marks is tricky since delimiters are exactly the same so opening and closing marks can not be distinguished from each other without taking a look at adjacent strings. What you can do is change a delimiter to something else so that you can look for it later.

Step 1:

Search for: "[^"\\]*(?:\\.[^"\\]*)*"

Replace with: $0Я

Step 2:

Search for: a(?=[^"\\]*(?:\\.[^"\\]*)*"Я)

Replace with whatever you expect.

Step 3:

Search for:

Replace with nothing to revert every thing.


Answers 2

Firstly a few of considerations:

  1. There could be multiple a characters within a single quote.
  2. Each quote (using single or double quotation marks) consists of an opening quote character, some text and the same closing quote character. A simple approach is to assume that when the quote characters are counted sequentially, the odd ones are opening quotes and the even ones are closing quotes.
  3. Following point 2, it could be worth some further thought on whether single-quoted strings should be allowed. See the following example: It's a shame 'this quoted text' isn't quoted. Here, the simple approach would think there were two quoted strings: s a shame and isn. Another: This isn't a quote ...'this is' and 'it's unclear where this quote ends'. I've avoided attempting to tackle these complexities and gone with the simple approach below.

The bad news is that point 1 presents a bit of a problem, as a capturing group with a wildcard repeat character after it (e.g. (.*)*) will only capture the last captured "thing". But the good news is there's a way of getting around this within certain limits. Many regex engines will allow up to 99 capturing groups (*). So if we can make the assumption that there will be no more than 99 as in each quote (UPDATE ...or even if we can't - see step 3), we can do the following...

(*) Unfortunately my first port of call, Notepad++ doesn't - it only allows up to 9. Not sure about VS Code. But regex101 (used for the online demos below) does.

TL;DR - What to do?

  1. Search for: "([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*([^a"]*)a*"
  2. Replace with: "\1\2\3\4\5\6\7\8\9\10\11\12\13\14\15\16\17\18\19\20\21\22\23\24\25\26\27\28\29\30\31\32\33\34\35\36\37\38\39\40\41\42\43\44\45\46\47\48\49\50\51\52\53\54\55\56\57\58\59\60\61\62\63\64\65\66\67\68\69\70\71\72\73\74\75\76\77\78\79\80\81\82\83\84\85\86\87\88\89\90\91\92\93\94\95\96\97\98\99"
  3. (Optionally keep repeating steps the previous two steps if there's a possibility of > 99 such characters in a single quote until they've all been replaced).
  4. Repeat step 1 but replacing all " with ' in the regular expression, i.e: '([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*([^a']*)a*'
  5. Repeat steps 2-3.

Online demos

Please see the following regex101 demos, which could actually be used to perform the replacements if you're able to copy the whole text into the contents of "TEST STRING":

Answers 3

/(["'])(.*?)(a)(.*?\1)/g 

With the replace pattern:

$1$2$4 

As far as I'm aware, VS Code uses the same regex engine as JavaScript, which is why I've written my example in JS.

The problem with this is that if you have multiple a's in 1 set of quotes, then it will struggle to pull out the right values, so there needs to be some sort of code behind it, or you, hammering the replace button until no more matches are found, to recurse the pattern and get rid of all the a's in between quotes

let regex = /(["'])(.*?)(a)(.*?\1)/g,  subst = `$1$2$4`,  str = `"a"  "helapke"  Not matched - aaaaaaa  "This is the way the world ends"  "Not with fire"  "ABBA"  "abba",  'I can haz cheezburger'  "This is not a match'  `;      // Loop to get rid of multiple a's in quotes  while(str.match(regex)){      str = str.replace(regex, subst);  }    const result = str;  console.log(result);

Answers 4

If you can use Visual Studio (instead of Visual Studio Code), it is written in C++ and C# and uses the .NET Framework regular expressions, which means you can use variable length lookbehinds to accomplish this.

(?<="[^"\n]*)a(?=[^"\n]*") 

Adding some more logic to the above regular expression, we can tell it to ignore any locations where there are an even amount of " preceding it. This prevents matches for a outside of quotes. Take, for example, the string "a" a "a". Only the first and last a in this string will be matched, but the one in the middle will be ignored.

(?<!^[^"\n]*(?:(?:"[^"\n]*){2})+)(?<="[^"\n]*)a(?=[^"\n]*") 

Now the only problem is this will break if we have escaped " within two double quotes such as "a\"" a "a". We need to add more logic to prevent this behaviour. Luckily, this beautiful answer exists for properly matching escaped ". Adding this logic to the regex above, we get the following:

(?<!^[^"\n]*(?:(?:"(?:[^"\\\n]|\\.)*){2})+)(?<="[^"\n]*)a(?=[^"\n]*") 

I'm not sure which method works best with your strings, but I'll explain this last regex in detail as it also explains the two previous ones.

  • (?<!^[^"\n]*(?:(?:"(?:[^"\\\n]|\\.)*){2})+) Negative lookbehind ensuring what precedes doesn't match the following
    • ^ Assert position at the start of the line
    • [^"\n]* Match anything except " or \n any number of times
    • (?:(?:"(?:[^"\\\n]|\\.)*){2})+ Match the following one or more times. This ensures if there are any " preceding the match that they are balanced in the sense that there is an opening and closing double quote.
      • (?:"(?:[^"\\\n]|\\.)*){2} Match the following exactly twice
      • " Match this literally
      • (?:[^"\\\n]|\\.)* Match either of the following any number of times
        • [^"\\\n] Match anything except ", \ and \n
        • \\. Matches \ followed by any character
  • (?<="[^"\n]*) Positive lookbehind ensuring what precedes matches the following
    • " Match this literally
    • [^"\n]* Match anything except " or \n any number of times
  • a Match this literally
  • (?=[^"\n]*") Positive lookahead ensuring what follows matches the following
    • [^"\n]* Match anything except " or \n any number of times
    • " Match this literally

You can drop the \n from the above pattern as the following suggests. I added it just in case there's some sort of special cases I'm not considering (i.e. comments) that could break this regex within your text. The \A also forces the regex to match from the start of the string (or file) instead of the start of the line.

(?<!\A[^"]*(?:(?:"(?:[^"\\]|\\.)*){2})+)(?<="[^"]*)a(?=[^"]*") 

You can test this regex here

This is what it looks like in Visual Studio:

Visual Studio example

Answers 5

I am using VSCode, but I'm open to any suggestions.

If you want to stay in an Editor environment, you could use
Visual Studio (>= 2012) or even notepad++ for quick fixup.
This avoids having to use a spurious script environment.

Both of these engines (Dot-Net and boost, respectively) use the \G construct.
Which is start the next match at the position where the last one left off.

Again, this is just a suggestion.

This regex doesn't check the validity of balanced quotes within the entire
string ahead of time (but it could with the addition of a single line).

It is all about knowing where the inside and outside of quotes are.

I've commented the regex, but if you need more info let me know.
Again this is just a suggestion (I know your editor uses ECMAScript).

Find (?s)(?:^([^"]*(?:"[^"a]*(?=")"[^"]*(?="))*"[^"a]*)|(?!^)\G)a([^"a]*(?:(?=a.*?")|(?:"[^"]*$|"[^"]*(?=")(?:"[^"a]*(?=")"[^"]*(?="))*"[^"a]*)))
Replace $1b$2

That's all there is to it.

https://regex101.com/r/loLFYH/1

Comments

(?s)                          # Dot-all inine modifier  (?:       ^                             # BOS        (                             # (1 start), Find first quote from BOS (written back)            [^"]*             (?:                           # --- Cluster                 " [^"a]*                      # Inside quotes with no 'a'                 (?= " )                 " [^"]*                       # Between quotes, get up to next quote                 (?= " )            )*                            # --- End cluster, 0 to many times             " [^"a]*                      # Inside quotes, will be an 'a' ahead of here                                          # to be sucked up by this match                  )                             # (1 end)     |                              # OR,        (?! ^ )                       # Not-BOS        \G                            # Continue where left off from last match.                                     # Must be an 'a' at this point  )  a                             # The 'a' to be replaced   (                             # (2 start), Up to the next 'a' (to be written back)       [^"a]*        (?:                           # --------------------            (?= a .*? " )                 # If stopped before 'a', must be a quote ahead         |                              # or,            (?:                           # --------------------                 " [^"]* $                     # If stopped at a quote, check for EOS              |                              # or,                  " [^"]*                       # Between quotes, get up to next quote                 (?= " )                  (?:                           # --- Cluster                      " [^"a]*                      # Inside quotes with no 'a'                      (?= " )                      " [^"]*                       # Between quotes                       (?= " )                 )*                            # --- End cluster, 0 to many times                  " [^"a]*                      # Inside quotes, will be an 'a' ahead of here                                               # to be sucked up on the next match                                )                             # --------------------       )                             # --------------------  )                             # (2 end) 

Answers 6

"Inside double quotes" is rather tricky, because there are may complicating scenarios to consider to fully automate this.

What are your precise rules for "enclosed by quotes"? Do you need to consider multi-line quotes? Do you have quoted strings containing escaped quotes or quotes used other than starting/ending string quotation?

However there may be a fairly simple expression to do much of what you want.

Search expression: ("[^a"]*)a

Replacement expression: $1b

This doesn't consider inside or outside of quotes - you have do that visually. But it highlights text from the quote to the matching character, so you can quickly decide if this is inside or not.

If you can live with the visual inspection, then we can build up this pattern to include different quote types and upper and lower case.

Read More

Friday, August 11, 2017

Json.obj Scala, string concat: Compilation error

Leave a Comment

I'm trying to do the next in Scala, I'm using play2:

val str = "another" val r = Json.obj("error_type" -> "invalid_request_error",             "validation_errors" -> (Json.obj(               "code" -> "this mode " + str + " does not exist",               "param" -> "mode"             ))) 

but it gives to me the error:

Type mismatch, expected: (String, Json.JsValueWrapper), actual: String 

but if I do:

val r = Json.obj("error_type" -> "invalid_request_error",             "validation_errors" -> (Json.obj(               ("this mode ".+(str)).+(" does not exist"),               "param" -> "mode"             )))) 

It compile and works...

How can I write it in the form str1 + str2 + str3 more readable? How is the order/precedence related here? In my answer I don't understand why is the () needed neither the comment. Is there another similar case when parenthesis are needed?

ps: I'm not sure if in Java is the same issue

3 Answers

Answers 1

This is easily explained by looking at operator precedence.

From the language reference http://scala-lang.org/files/archive/spec/2.11/06-expressions.html#infix-operations, we can see that operators + and -> have the same precedence. This is because, in general, it is the first character of an operator that determines its precedence. In our case, the first characters are + and -, which both have the same precedence.

thus, writing "code" -> "this mode " + str + " does not exist" is the same as writing:

"code"   .->("this mode ")   .+(str)   .+(" does not exist") 

This is consistent with what the compiler tells you:

  • the result type of the first operation ("code" -> "this mode ") is (String, String) which is equivalent to Tuple2[String, String]
  • (String, String) + String triggers an implicit toString() conversion on the tuple, therefore the resulting type is String.

You seem to have already found the better way to format it in a more readable way.

As to other cases were parentheses are needed, the obvious answer would be that you need them as soon as you don't want what the behavior that operator precedence would give you. As such I highly recommend reading chapter 6.12 of the spec linked above!

Answers 2

Finally I could do it, but I don't know the reason, I someone know, please let me know:

I sorrowed the strings with () and it compiled and works like a charm:

"code" -> ("payment mode " + another + " does not exist"), ... 

all together it would be:

Json.obj("error_type" -> "invalid_request_error",                       "validation_errors" -> (Json.obj(                         "code" -> ("payment mode " + another + " does not exist"),                         "param" -> "payment_mode"                       )))) 

Answers 3

You can create your error messages in a map of String and Seq[String] and then transform them into Json. I think that would be the best way to do it.

Read More

Monday, June 12, 2017

Finding the longest border of a string

Leave a Comment

First, let me tell you what the border of a string is,

let x = "abacab" let y = "ababab" 

The border of a string is a substring which is both a proper prefix and proper suffix of the string — "proper" meaning that the whole string does not count as a substring. The longest border of x is "ab". The longest border of y is "abab" (the prefix and suffix can overlap).

Another example:
In string "abcde hgrab abcde", then "abcde" is a prefix as well as suffix. Thus it is also the longest border of the string above.

How can I find the longest border of a string?

9 Answers

Answers 1

x = abacab y = String.reverse(x)  border = ""  if (x == y) {   print border;   return; }   for (int i = 0; i < x.length; i++) {   if (x[i] == y[i])     border += x[i]   else     break }  print border 

Answers 2

Finding the "border of a string" is what the prefix function (also known as failure function) of Knuth-Morris-Pratt algorithm do. Implementation in c++ (a bit changed version of this code):

int longestBorder(const string& s) {     int len = s.length();     vector<int> prefixFunc(len);      prefixFunc[0] = 0;      int curBorderLen = 0;        for (int i = 1; i < len; ++i) {          while (curBorderLen > 0 && s[curBorderLen] != s[i])              curBorderLen = prefixFunc[curBorderLen - 1];           if (s[curBorderLen] == s[i])              ++curBorderLen;          prefixFunc[i] = curBorderLen;     }      return prefixFunc[len-1]; } 

Runnable version: http://ideone.com/hTW8FL

The complexity of this algorithm is O(n).

Answers 3

Here's a Java implementation, based on the assumption that borders are proper substrings. (Otherwise the longest border is simply the string length.)

public static int findLongestBorder(String s) {     int len = s.length();     for (int i = len - 1; i > 0; i--) {         String prefix = s.substring(0, i);         String suffix = s.substring(len - i, len);         if (prefix.equals(suffix)) {             return i;         }     }     return 0; } 

This could be optimized a bit by starting with the string's character array and then comparing individual characters, but the idea behind the algorithm is clearer the way I wrote it.

Answers 4

Here is a JS solution with commentary that uses the prefix function that DAIe mentioned:

function getPrefixBorders(string) {     // This will contain the border length for each     // prefix in ascending order by prefix length.     var borderLengthByPrefix = [0];      // This is the length of the border on the current prefix.     var curBorderLength = 0;      // Loop from the 2nd character to the last.     for (var i = 1; i < string.length; i++) {          // As long as a border exists but the character         // after it doesn't match the current character,          while (curBorderLength > 0 && string[curBorderLength] !== string[i])             // set the border length to the length of the current border's border.             curBorderLength = borderLengthByPrefix[curBorderLength - 1];          // If the characters do match,         if (string[curBorderLength] === string[i])             // the new border is 1 character longer.             curBorderLength++;          // Note the border length of the current prefix.         borderLengthByPrefix[i] = curBorderLength;     }      return borderLengthByPrefix; } 

It returns the longest border lengths of every prefix in a string (which is a lot more than asked for, but it does so in linear time). So to get the length of the longest border in the full string:

var string = "ababab"; var borderLengthsByPrefix = getPrefixBorders(); // [0,0,1,2,3,4] var stringBorderLength = borderLengthsByPrefix[borderLengthsByPrefix.length - 1]; 

Another great resource for understanding how this works is this video (and the one before it) on Coursera.

Answers 5

To get the length of the longest border, do this:

def get_border_size(astr):     border = 0     for i in range(len(astr)):         if astr[:i] == astr[-i:]:             border = i     return border 

To get the longest border itself, this:

def get_border(astr):     border = 0     for i in range(len(astr)):         if astr[:i] == astr[-i:]:             border = astr[:i]     return border 

Answers 6

I've made a solution using Python3 (works also with Python2), using Counter from collections module and max().

Here is my solution:

from collections import Counter  def get_seq(a):     data = []     for k in range(1, len(a)):         data.append(a[:k])         data.append(a[k:])      return Counter(data)  def get_max_sublist(a):     bb = [k for k in a.items() if k[1] > 1]     try:         k, j = max(bb, key= lambda x: len(x[0]))         n, _ = max(a.items(), key= lambda x: x[1])      except ValueError:         return None      else:         return k if j > 1 else n    seq = ["abacab", "ababab", "abxyab", "abxyba", "abxyzf", "bacab"]  for k in seq:     j = get_seq(k)     print("longest border of {} is: {}".format(k, get_max_sublist(j))) 

Output:

longest border of abacab is: ab longest border of ababab is: abab longest border of abxyab is: ab longest border of abxyba is: a longest border of abxyzf is: None longest border of bacab is: b 

Answers 7

This simple solution with a single loop works just fine:

function findLongestBorder($s){     $a = 0;     $b = 1;     $n = strlen($s);      while($b<$n){         if($s[$a]==$s[$b]){             $a++;         }else{             $b-= $a;             $a = 0;         }         $b++;     }      return substr($s,0,$a); } 

Example:

echo findLongestBorder("abacab")."\n"; echo findLongestBorder("ababab")."\n"; echo findLongestBorder("abcde hgrab abcde")."\n"; echo findLongestBorder("bacab")."\n"; echo findLongestBorder("abacababac")."\n"; 

Output:

ab abab abcde b abac 

See https://eval.in/812640

Answers 8

I've been using a lot of javascript lately so I did it with Javascript:

function findBorder() {    var givenString = document.getElementById("string").value;    var length = givenString.length;    var y = length;    var answer;    var subS1;    var subS2;    for (var x = 0; x < length; x++ ){      subS1 = givenString.substring(0, x);      subS2 = givenString.substring(y);      if(subS2 === subS1){        answer = subS1;      }      y--;    }    document.getElementById("answer").innerHTML = answer.toString();  }
<h1>put the string in here</h1>    <input type="text" id="string" />  <button id="goButton" onclick="findBorder()">GO</button>      <h3 id="answer"></h3>

Answers 9

If you are talking about character arrays, I think you want the following. This is based on the border being the first and last character of a string. Your examples aren't clear as to what a border is. You need to more clearly define what a border is.

x = abcde border = { x[0], x[length(x)-1) } 

and if you need length

length(z) {     return sizeof(z) / (sizeof(z[0]) 
Read More

Wednesday, May 3, 2017

How to find set of shortest subsequences with minimal collisions from set of strings

Leave a Comment

I've got a list of strings like

  • Foobar
  • Foobaron
  • Foot
  • barstool
  • barfoo
  • footloose

I want to find the set of shortest possible sub-sequences that are unique to each string in the set; the characters in each sub-sequence do not need to be adjacent, just in order as they appear in the original string. For the example above, that would be (along other possibilities)

  • Fb (as unique to Foobar as it gets; collision with Foobaron unavoidable)
  • Fn (unique to Foobaron, no other ...F...n...)
  • Ft (Foot)
  • bs (barstool)
  • bf (barfoo)
  • e (footloose)

Is there an efficient way to mine such sequences and minimize the number of colliding strings (when collisions can't be avoided, e.g. when strings are substrings of other strings) from a given array of strings? More precisely, chosing the length N, what is the set of sub-sequences of up to N characters each that identify the original strings with the fewest number of collisions.

3 Answers

Answers 1

I would'nt really call that 'efficient', but you can do better than totally dumb like that:

words = ['Foobar', 'Foobaron', 'Foot', 'barstool', 'barfoo', 'footloose'] N = 2 n = len(words) L = max([len(word) for word in words])  def generate_substrings(word, max_length=None):     if max_length is None:         max_length = len(word)     set_substrings = set()     set_substrings.add('')     for charac in word:         new_substr_list = []         for substr in set_substrings:             new_substr = substr + charac             if len(new_substr) <= max_length:                 new_substr_list.append(new_substr)         set_substrings.update(new_substr_list)     return set_substrings  def get_best_substring_for_each(string_list=words, max_length=N):     all_substrings = {}     best = {}     for word in string_list:         for substring in generate_substrings(word, max_length=max_length):             if substring not in all_substrings:                 all_substrings[substring] = 0             all_substrings[substring] = all_substrings[substring] + 1     for word in string_list:         best_score = len(string_list) + 1         best[word] = ''         for substring in generate_substrings(word=word, max_length=max_length):             if all_substrings[substring] < best_score:                 best[word] = substring                 best_score = all_substrings[substring]     return best  print(get_best_substring_for_each(words, N)) 

This program prints the solution:

{'barfoo': 'af', 'Foobar': 'Fr', 'Foobaron': 'n', 'footloose': 'os', 'barstool': 'al', 'Foot': 'Ft'} 

This can still be improved easily by a constant factor, for instance by storing the results of generate_substringsinstead of computing it twice.

The complexity is O(n*C(N, L+N)), where n is the number of words and L the maximum length of a word, and C(n, k) is the number of combinations with k elements out of n.

I don't think (not sure though) that you can do much better in the worst case, because it seems hard not to enumerate all possible substrings in the worst case (the last one to be evaluated could be the only one with no redundancy...). Maybe in average you can do better...

Answers 2

You could use a modification to the longest common subsequence algorithm. In this case you are seeking the shortest unique subsequence. Shown below is part of a dynamic programming solution which is more efficient than a recursive solution. The modifications to the longest common subsequence algorithm are described in the comments below:

for (int i = 0; i < string1.Length; i++)     for (int j = 0; j < string2.Length; j++)         if (string1[i-1] != string2[j-1])   // find characters in the strings that are distinct             SUS[i][j] = SUS[i-1][j-1] + 1;  // SUS: Shortest Unique Substring         else             SUS[i][j] = min(SUS[i-1][j], SUS[i][j-1]);  // find minimum size of distinct strings 

You can then put this code in a function and call this function for each string in your set to find the length of the shortest unique subsequence in the set.

Once you have the length of the shortest unique subsequence, you can backtrack to print the subsequence.

Answers 3

You should use modified Trie structure, insert strings to a trie in a way that :

Foo-bar-on    -t bar-stool    -foo 

The rest is straightforward, just choose correct compressed node[0] char

That Radix tree should help

Read More

Wednesday, April 19, 2017

How to find set of shortest subsequences with minimal collisions from set of strings

Leave a Comment

I've got a list of strings like

  • Foobar
  • Foobaron
  • Foot
  • barstool
  • barfoo
  • footloose

I want to find the set of shortest possible sub-sequences that are unique to each string in the set; the characters in each sub-sequence do not need to be adjacent, just in order as they appear in the original string. For the example above, that would be

  • Fb (as unique to Foobar as it gets; collision with Foobaron unavoidable)
  • Fn (unique to Foobaron, no other ...F...n...)
  • Ft (Foot)
  • bs (barstool)
  • bf (barfoo)
  • e (footloose)

Is there an efficient way to mine such sequences and minimize the number of colliding strings (when collisions can't be avoided, e.g. when strings are substrings of other strings) from a given array of strings? More precisely, chosing the length N, what is the set of sub-sequences of up to N characters each that identify the original strings with the fewest number of collisions.

1 Answers

Answers 1

I would'nt really call that 'efficient', but you can do better than totally dumb like that:

words = ['Foobar', 'Foobaron', 'Foot', 'barstool', 'barfoo', 'footloose'] N = 2 n = len(words) L = max([len(word) for word in words])  def generate_substrings(word, max_length=None):     if max_length is None:         max_length = len(word)     set_substrings = set()     set_substrings.add('')     for charac in word:         new_substr_list = []         for substr in set_substrings:             new_substr = substr + charac             if len(new_substr) <= max_length:                 new_substr_list.append(new_substr)         set_substrings.update(new_substr_list)     return set_substrings  def get_best_substring_for_each(string_list=words, max_length=N):     all_substrings = {}     best = {}     for word in string_list:         for substring in generate_substrings(word, max_length=max_length):             if substring not in all_substrings:                 all_substrings[substring] = 0             all_substrings[substring] = all_substrings[substring] + 1     for word in string_list:         best_score = len(string_list) + 1         best[word] = ''         for substring in generate_substrings(word=word, max_length=max_length):             if all_substrings[substring] < best_score:                 best[word] = substring                 best_score = all_substrings[substring]     return best  print(get_best_substring_for_each(words, N)) 

This program prints the solution:

{'barfoo': 'af', 'Foobar': 'Fr', 'Foobaron': 'n', 'footloose': 'os', 'barstool': 'al', 'Foot': 'Ft'} 

This can still be improved easily by a constant factor, for instance by storing the results of generate_substringsinstead of computing it twice.

The complexity is O(n*C(N, L+N)), where n is the number of words and L the maximum length of a word, and C(n, k) is the number of combinations with k elements out of n.

I don't think (not sure though) that you can do much better in the worst case, because it seems hard not to enumerate all possible substrings in the worst case (the last one to be evaluated could be the only one with no redundancy...). Maybe in average you can do better...

Read More