Display a python output with special characters in GUI label unity

Am running a python script from unity, and I want the output to be displayed in a GUI label in unity, the output contain special characters like é,ç,... but am getting symbols instead. This is what I tried so far but no luck

using (Process proc = Process.Start (start)) {             using (StreamReader reader = proc.StandardOutput) {                 string result = reader.ReadToEnd ();                 UTF8Encoding utf8 = new UTF8Encoding ();                 byte[] encodedBytes = utf8.GetBytes (result);                 UnityEngine.Debug.Log (result);                 Message1.text = result;              }         } 

result is the output: it should be

('fromaje', 'Fausse:') ('Correction suggérér:','fromage') 

But am getting:

1 Answers

Answers 1

When you create your process use:

    var proc = new Process {     StartInfo = new ProcessStartInfo {         FileName = "python.exe whatever.py",         Arguments = "yourargs",         UseShellExecute = false,         RedirectStandardOutput = true,         CreateNoWindow = true     }  }; 

Then read the string by doing this:

proc.Start(); StringBuilder bldr = new StringBuilder(); while (!proc.StandardOutput.EndOfStream) {     bldr.append(proc.StandardOutput.ReadLine()+"\n"); } 

You can get the full output by doing:

bldr.ToString() 

Read More

How to convert a file to UTF-8 in php?

Is it possible to convert a file into UTF-8 on my end?

If I have an access on the file after the submission with

$_FILES['file']['tmp_name'] 

Note: The user can upload a CSV file with any kind of charset, I usually encounter an unknown 8-bit charset.

I try

$row = array(); $datas = file($_FILES['file']['tmp_name']); foreach($datas as $data) {     $data = mb_convert_encoding($data, 'UTF-8');     $row[] = explode(',', $data); } 

But the problem is, this code remove special characters like single quote.

My first question is htmlspecialchars remove the value inside the array?

I put it for additional information. Thanks for those who can help!

4 Answers

Answers 1

Try this out.
The example I have used was something I was doing in a test environment, you might need to change the code slightly.

I had a text file with the following data in:

test café áÁÁÁááá žžœš¥± ÆÆÖÖÖasØØ ß 

Then I had a form which took a file input in and performed the following code:

function neatify_files(&$files) {     $tmp = array();     for ($i = 0; $i < count($_FILES); $i++) {         for ($j = 0; $j < count($_FILES[array_keys($_FILES)[$i]]["name"]); $j++) {             $tmp[array_keys($_FILES)[$i]][$j]["name"] = $_FILES[array_keys($_FILES)[$i]]["name"][$j];             $tmp[array_keys($_FILES)[$i]][$j]["type"] = $_FILES[array_keys($_FILES)[$i]]["type"][$j];             $tmp[array_keys($_FILES)[$i]][$j]["tmp_name"] = $_FILES[array_keys($_FILES)[$i]]["tmp_name"][$j];             $tmp[array_keys($_FILES)[$i]][$j]["error"] = $_FILES[array_keys($_FILES)[$i]]["error"][$j];             $tmp[array_keys($_FILES)[$i]][$j]["size"] = $_FILES[array_keys($_FILES)[$i]]["size"][$j];         }     }     return $files = $tmp; }  if (isset($_POST["submit"])) {     neatify_files($_FILES);     $file = $_FILES["file"][0];      $handle = fopen($file["tmp_name"], "r");     while ($line = fgets($handle)) {         $enc = mb_detect_encoding($line, "UTF-8", true);         if (strtolower($enc) != "utf-8") {             echo "<p>" . (iconv($enc, "UTF-8", $line)) . "</p>";         } else {             echo "<p>$line</p>";         }     } } ?> <form action="<?= $_SERVER["PHP_SELF"]; ?>" method="POST" enctype="multipart/form-data">     <input type="file" name="file[]" />     <input type="submit" name="submit" value="Submit" /> </form> 

The function neatify_files is something I wrote to make the $_FILES array more logical in its layout.

The form is a standard form that simply POSTs the data to the server.
Note: Using $_SERVER["PHP_SELF"] is a security risk, see here for more.

When the data is posted I store the file in a variable. Obviously, if you are using the multiple attribute your code won't look quite like this.

$handle stores the entire contents of the text file, in a read-only format; hence the "r" argument.

$enc uses the mb_detect_encoding function to detect the encoding (duh).
At first I was having trouble with obtaining the correct encoding. Setting the encoding_list to use only UTF-8, and setting strict to be true.

If the encoding is UTF-8 then I simply print the line, if it didn't I converted it to UTF-8 using the iconv function.

Answers 2

before you can convert it to utf-8, you need to know what characterset it is. if you can't figure that out, you can't in any sane way convert it to utf8.. however, an insane way to convert it to utf-8, if the encoding cannot be determined, is to simply strip any bytes that doesn't happen to be valid in utf-8, you might be able to use that as a fallback...

warning, untested code (im suddenly in a hurry), but may look something like this:

foreach ( $datas as $data ) {     $encoding = guess_encoding ( $data );     if (empty ( $encoding )) {         // encoding cannot be determined...         // as a fallback, we simply strip any bytes that isnt valid utf-8...         // obviously this isn't a reliable conversion scheme.         // also this could probably be improved         $data = iconv ( "ASCII", "UTF-8//TRANSLIT//IGNORE", $text );     } else {         $data = mb_convert_encoding ( $data, 'UTF-8', $encoding );     }     $row [] = explode ( ',', $data ); } function guess_encoding(string $str): string {     $blacklist = array (             'pass',             'auto',             'wchar',             'byte2be',             'byte2le',             'byte4be',             'byte4le',             'BASE64',             'UUENCODE',             'HTML-ENTITIES',             '7bit',             '8bit'      );     $encodings = array_flip ( mb_list_encodings () );     foreach ( $blacklist as $tmp ) {         unset ( $encodings [$tmp] );     }     $encodings = array_keys ( $encodings );     $detected = mb_detect_encoding ( $str, $encodings, true );     return ( string ) $detected; } 

Answers 3

you can convert the file text into binary data by using the following

FUNCTION bin2text($bin_str)  {      $text_str = '';      $chars = EXPLODE("\n", CHUNK_SPLIT(STR_REPLACE("\n", '', $bin_str), 8));      $_I = COUNT($chars);      FOR($i = 0; $i < $_I; $text_str .= CHR(BINDEC($chars[$i])), $i  );      RETURN $text_str;  }   FUNCTION text2bin($txt_str)  {      $len = STRLEN($txt_str);      $bin = '';      FOR($i = 0; $i < $len; $i  )      {          $bin .= STRLEN(DECBIN(ORD($txt_str[$i]))) < 8 ? STR_PAD(DECBIN(ORD($txt_str[$i])), 8, 0, STR_PAD_LEFT) : DECBIN(ORD($txt_str[$i]));      }      RETURN $bin;  } 

after converting the data into binary you simply change the text to php method mb_convert_encoding($fileText, "UTF-8");

Answers 4

Let's try this:

function encode_utf8($data) {     if ($data === null || $data === '') {         return $data;     }     if (!mb_check_encoding($data, 'UTF-8')) {         return mb_convert_encoding($data, 'UTF-8');     } else {         return $data;     } } 

Usage:

$content = file_get_contents($_FILES['file']['tmp_name']); $content = encode_utf8($content);  $rows = explode("\n", $content); foreach ($rows as $row) {     print_r($row); } 

Read More

Uppercase surname, excluding the lowercase prefix section of a surname

I am trying to determine a method to uppercase a surname; however, excluding the lowercase prefix.

Example of names and their conversion:

• MacArthur -> MacARTHUR
• McDavid -> McDAVID
• LeBlanc -> LeBLANC
• McIntyre -> McINTYRE
• de Wit -> de WIT

There are also names that would contain the surnames that would need to be fully capitalized, so a simple function to identify the prefix such as strchr()would not suffice:

• Macmaster -> MACMASTER
• Macintosh -> MACINTOSH

The PHP function mb_strtoupper() is not appropriate, as it capitalizes the complete string. Similarly strtoupper() is not appropriate, and loses accents on accented names as well.

There are some answers around SO that partly answer the question, such as : Capitalization using PHP However, the common shortfall is assuming that all names with a surname as as Mac are followed with a capital.

The names are capitalized properly in the database, so we can assume that a name spelled as Macarthur is correct, or MacArthur is correct for another person.

7 Answers

Answers 1

Going with the rule to capitalise everything after the last capital letter:

preg_replace_callback('/\p{Lu}\p{Ll}+$/u',                        function ($m) { return mb_strtoupper($m[0]); },                       $name) 

\p{Lu} and \p{Ll} are Unicode upper and lower case characters respectively, and mb_strtoupper is unicode aware… for a simple ASCII-only variant this would do too:

preg_replace_callback('/[A-Z][a-z]+$/',                        function ($m) { return strtoupper($m[0]); },                       $name) 

Answers 2

I believe this is the solution to question:

$names = array(     'MacArthur',     'Macarthur',     'ÜtaTest',     'de Wit' );  $pattern = '~(?<prefix>(?:\p{Lu}.+|.+\s+))(?<suffix>\p{Lu}.*)~'; foreach ($names as $key => $name) {     if (preg_match($pattern, $name, $matches)) {         $names[$key] = $matches['prefix'] . mb_strtoupper($matches['suffix']);     } else {         $names[$key] = mb_strtoupper($name);     } }  print_r($names); 

it produces following result for the input array above:

Array (     [0] => MacARTHUR     [1] => MACARTHUR     [2] => ÜtaTEST     [3] => de WIT ) 

Brief explanation of regular expression:

(?<prefix>             # name of the captured group    (?:                 # ignore this group        \p{Lu}.+        # any uppercase character followed by any character        |               # OR        .+\s+           # any character followed by white space    ) ) (?<suffix>             # name of the captured group     \p{Lu}.*           # any uppercase character followed by any character ) 

Answers 3

Here's a basic algorithm that avoids cryptic regular expressions:

1. Create a multibyte-safe character array for the literal surname (as it exists in the database).
2. Create a second character array in multibyte-safe capitalized form.
3. Intersect both arrays to determine the index of the final capitalized character.
4. Concatenate the literal surname through the index with the capitalized form after the index.

In code form:

<?php $names = [     'MacArthur',     'McDavid',     'LeBlanc',     'McIntyre',     'de Wit',     'Macmaster',     'Macintosh',     'MacMac',     'die Über',     'Van der Beek',     'johnson',     'Lindström',     'Cehlárik', ];  // Uppercase after the last capital letter function normalizeSurname($name) {     // Split surname into a Unicode character array     $chars = preg_split('//u', $name, -1, PREG_SPLIT_NO_EMPTY);      // Capitalize surname and split into a character array     $name_upper = mb_convert_case($name, MB_CASE_UPPER);     $chars_upper = preg_split('//u', $name_upper, -1, PREG_SPLIT_NO_EMPTY);      // Find the index of the last capitalize letter     @$last_capital_idx = array_slice(array_keys(array_intersect($chars, $chars_upper)), -1)[0] ?: 0;      // Concatenate the literal surname up to the index, and capitalized surname thereafter     return mb_substr($name, 0, $last_capital_idx) . mb_substr($name_upper, $last_capital_idx); }  // Loop through the surnames and display in normalized form foreach($names as $name) {     echo sprintf("%s -> %s\n",          $name,         normalizeSurname($name)     ); } 

You'll get output like:

MacArthur -> MacARTHUR McDavid -> McDAVID LeBlanc -> LeBLANC McIntyre -> McINTYRE de Wit -> de WIT Macmaster -> MACMASTER Macintosh -> MACINTOSH MacMac -> MacMAC die Über -> die ÜBER Van der Beek -> Van der BEEK johnson -> JOHNSON Lindström -> LINDSTRÖM Cehlárik -> CEHLÁRIK 

This makes the assumption that an entirely lowercase surname should be capitalized. It would be easy to change that behavior.

Answers 4

  $string = "McBain";   preg_match('/([A-Z][a-z]+\h*)$/', $string, $matches);   /**     Added qualifier for if no match found    **/   if(!empty($matches[1])){       // $upperString = str_replace($matches[1], strtoupper($matches[1]),$string);       // replace only last occurance of string:       $pos = strrpos($string, $matches[1]);      if($pos !== false)          {          $upperString = substr_replace($string, strtoupper($matches[1]), $pos, strlen($matches[1]));           }   }   else {       $upperString = strtoupper($string);   }   print $upperString; 

Example Output:

$string = "McBain "; $upperString = "McBAIN";  $string = "Mac Hartin"; $upperString = "Mac HARTIN";  $string = "Macaroni "; $upperString = "MACARONI";  $string = "jacaroni"; $upperString = "JACARONI";  $string = "MacMac"; $upperString = "MacMAC"; 

( Also added a \h* to the regex to catch any whitespace. )

reference for find/replace last occurance.

Answers 5

<?php $string = "MacArthur"; $count = 0; $finished = ""; $chars = str_split($string); foreach($chars as $char){     if(ctype_upper($char)){         $count++;     }         if($count == 2){           $finished .= strtoupper($char);          }          else{           $finished .= $char;               }  }  echo $finished;  

Answers 6

Here is the code to uppercase all symbols after a last uppercase in the string.

preg_replace_callback('/[A-Z][^A-Z]+$/', function($match) {   return strtoupper($match[0]); }, $str); 

Try it with test examples from your question: https://repl.it/NYcR/5

Answers 7

Just to differ from the rest of the answers you could try something like this.

$names = array(     'MacArthur',     'Macarthur',     'ÜtaTest',     'de Wit' ); function fixSurnameA($item) { $lname = mb_strtolower($item); $nameArrayA = str_split($item,1); $nameArrayB = str_split($lname,1); $result = array_diff($nameArrayA, $nameArrayB); $keys = array_keys($result); $key = max($keys); if(count($keys)>=2 or (count($keys)==1 and $key>0)) { $pre = substr($item, 0, $key); $suf = mb_strtoupper(substr($item, $key)); echo $pre.$suf."\n"; } else {  echo $item."\n"; } } function fixSurnameB($item) { $lname = mb_strtolower($item); $nameArrayA = str_split($item,1); $nameArrayB = str_split($lname,1); $result = array_diff($nameArrayA, $nameArrayB); $keys = array_keys($result); $key = max($keys); $pre = substr($item, 0, $key); $suf = mb_strtoupper(substr($item, $key)); echo $pre.$suf."\n"; }  array_walk($names,'fixSurnameA'); /* MacARTHUR    Macarthur    ÜtaTEST    de WIT  */ array_walk($names,'fixSurnameB'); /* MacARTHUR    MACARTHUR    ÜtaTEST    de WIT  */ 

Test this on PHP SandBox

Read More

How to read the Content Type header and convert into utf-8 while Gmail IMAP has utf8 and Outlook has ISO-8859-7?

So I get emails using imap from gmail and outlook.

Gmail encodes like this =?UTF-8?B?UmU6IM69zq3OvyDOtc68zrHOuc67IG5ldyBlbWFpbA==?= and outlook encodes like this =?iso-8859-7?B?UmU6IOXr6+ft6er8IHN1YmplY3Q=?=

Unfortunately I did not find yet any solution that will help me make this into readable text. Instead I am messing with:

mb_convert_encoding($body, "UTF-8", "UTF-8");  

and

mb_convert_encoding($body, "UTF-8", "iso-8859-7"); 

but I am struggling to find a solution to solve this matter.

This is how I open the IMAP of my account (which has a lot of gmail and outlook messages)

$hostname = '{imappro.zoho.com:993/imap/ssl}INBOX'; $username = 'email@email.com'; $password = 'password';   /* try to connect */ $inbox = imap_open($hostname,$username ,$password) or die('Cannot connect to Zoho: ' . imap_last_error());  /* grab emails */ $emails = imap_search($inbox,'UNSEEN'); 

Any help?

4 Answers

Answers 1

Unfortunately I did not find yet any solution that will help me make this into readable text.

Solution Your strings are base64 encoded.

=?UTF-8?B?UmU6IM69zq3OvyDOtc68zrHOuc67IG5ldyBlbWFpbA==?=

echo base64_decode('UmU6IM69zq3OvyDOtc68zrHOuc67IG5ldyBlbWFpbA=='); 

prints "Re: νέο εμαιλ new email"

=?iso-8859-7?B?UmU6IOXr6+ft6er8IHN1YmplY3Q=?=

echo base64_decode('UmU6IOXr6+ft6er8IHN1YmplY3Q='); 

prints out "Re: subject"

The answer is to use base64_decode in conjunction with your current solutions.

The way to identify base64 encoded text is that it's depicted as letters a-z, A-Z, numbers 0-9 along with two other characters (usually + and /) and it's usually right padded with =.

Answers 2

look here

   /* connect to gmail */     $hostname = '{imap.gmail.com:993/imap/ssl}INBOX';     $username = 'davidwalshblog@gmail.com';     $password = 'davidwalsh';      /* try to connect */     $inbox = imap_open($hostname,$username,$password) or die('Cannot connect to Gmail: ' . imap_last_error());      /* grab emails */     $emails = imap_search($inbox,'ALL');      /* if emails are returned, cycle through each... */     if($emails) {          /* begin output var */         $output = '';          /* put the newest emails on top */         rsort($emails);          /* for every email... */         foreach($emails as $email_number) {              /* get information specific to this email */             $overview = imap_fetch_overview($inbox,$email_number,0);             $message = imap_fetchbody($inbox,$email_number,2);              /* output the email header information */             $output.= '<div class="toggler '.($overview[0]->seen ? 'read' : 'unread').'">';             $output.= '<span class="subject">'.$overview[0]->subject.'</span> ';             $output.= '<span class="from">'.$overview[0]->from.'</span>';             $output.= '<span class="date">on '.$overview[0]->date.'</span>';             $output.= '</div>';              /* output the email body */             $output.= '<div class="body">'.$message.'</div>';         }          echo $output;     }       /* close the connection */     imap_close($inbox); 

for reading and decoding look here

<?php $hostname = '{********:993/imap/ssl}INBOX'; $username = '*********'; $password = '******';  $inbox = imap_open($hostname,$username,$password) or die('Cannot connect to server: ' . imap_last_error());  $emails = imap_search($inbox,'ALL');  if($emails) {     $output = '';     rsort($emails);      foreach($emails as $email_number) {         $overview = imap_fetch_overview($inbox,$email_number,0);         $structure = imap_fetchstructure($inbox, $email_number);          if(isset($structure->parts) && is_array($structure->parts) && isset($structure->parts[1])) {             $part = $structure->parts[1];             $message = imap_fetchbody($inbox,$email_number,2);              if($part->encoding == 3) {                 $message = imap_base64($message);             } else if($part->encoding == 1) {                 $message = imap_8bit($message);             } else {                 $message = imap_qprint($message);             }         }          $output.= '<div class="toggle'.($overview[0]->seen ? 'read' : 'unread').'">';         $output.= '<span class="from">From: '.utf8_decode(imap_utf8($overview[0]->from)).'</span>';         $output.= '<span class="date">on '.utf8_decode(imap_utf8($overview[0]->date)).'</span>';         $output.= '<br /><span class="subject">Subject('.$part->encoding.'): '.utf8_decode(imap_utf8($overview[0]->subject)).'</span> ';         $output.= '</div>';          $output.= '<div class="body">'.$message.'</div><hr />';     }      echo $output; }  imap_close($inbox); ?> 

Look here for great tutorial on email structure, and function to extract it.

Answers 3

If you want to decode header elements, there is a PHP function for that: imap_mime_header_decode().

Also, you will need some MIME parser class to decode multipart messages.

Answers 4

To get the headers, you would pass your stream ($inbox) to imap_headers(). There are lots of values you can get in the response, full list: imap_headerinfo

For the actual messages, plain text can be read using imap_body(), passing the stream and the number of the message you want (in $emails after your search). Getting an html/multipart email is a bit trickier. First you need imap_fetchstructure(), which identifies the parts of the message, then imap_fetchbody() to get the piece you are interested in.

Once you have a result from imap_fetchbody(), if you still need to adjust the encoding, it could be done at this point.

Read More

UTF-8 / Unicode Text Encoding with RPostgreSQL

I'm running R on a Windows machine which is directly linked to a PostgreSQL database. I'm not using RODBC. My database is encoded in UTF-8 as confirmed by the following R command:

dbGetQuery(con, "SHOW CLIENT_ENCODING") #   client_encoding # 1            UTF8 

However, when some text is read into R, it displays as strange text in R.

For example, the following text is shown in my PostgreSQL database: "Stéphane"

After exporting to R it's shown as: "Stéphane" (the é is encoded as é)

When importing to R I use the dbConnect command to establish a connection and the dbGetQuery command to query data using SQL. I do not specify any text encoding anywhere when connecting to the database or when running a query.

I've searched online and can't find a direct resolution to my issue. I found this link, but their issue is with RODBC, which I'm not using.

This link is helpful in identifying the symbols, but I don't just want to do a find & replace in R... way too much data.

I did try running the following commands below and I arrived at a warning.

Sys.setlocale("LC_ALL", "en_US.UTF-8") # [1] "" # Warning message: # In Sys.setlocale("LC_ALL", "en_US.UTF-8") : #   OS reports request to set locale to "en_US.UTF-8" cannot be honored Sys.setenv(LANG="en_US.UTF-8") Sys.setenv(LC_CTYPE="UTF-8") 

The warning occurs on the Sys.setlocale("LC_ALL", "en_US.UTF-8") command. My intuition is that this is a Windows specific issue and doesn't occur with Mac/Linux/Unix.

EDIT 2014-01-29:
The following execution will fix any Unicode/UTF-8 problems in Windows. It must be executed before querying the database.

postgresqlpqExec(con, "SET client_encoding = 'windows-1252'") 

2 Answers

Answers 1

After exporting to R it's shown as: "Stéphane" (the é is encoded as é)

Your R environment is using a 1-byte non-composed encoding like latin-1 or windows-1252. Witness this test in Python, demonstrating that the utf-8 bytes for é, decoded as if they were latin-1, produce the text you see:

>>> print u"é".encode("utf-8").decode("latin-1") é 

Either SET client_encoding = 'windows-1252' or fix the encoding your R environment uses. If it's running in a cmd.exe console you'll need to mess with the chcp console command; otherwise it's specific to whatever your R runtime is.

Answers 2

As Craig Ringer said, setting client_encoding to windows-1252 is probably not the best thing to do. Indeed, if the data you're retrieving contains a single exotic character, you're in trouble:

Error in postgresqlExecStatement(conn, statement, ...) : RS-DBI driver: (could not Retrieve the result : ERROR: character 0xcca7 of encoding "UTF8" has no equivalent in "WIN1252" )

On the other hand, getting your R environment to use Unicode could be impossible (I have the same problem as you with Sys.setlocale... Same in this question too.).

A (dirty) workaround is to manually declare UTF-8 encoding on all your data, using a function like this one:

set_utf8 = function(x){   # Declare UTF-8 encoding on all character strings:   for (i in 1:ncol(x)){     if (is.character(x[, i])) Encoding(x[, i]) <- 'UTF-8'   }   # Same on column names:   for (name in colnames(x)){     Encoding(name) <- 'UTF-8'   }   return(x) } 

And you have to use this function in all your queries:

set_utf8(dbGetQuery(con, "SELECT myvar FROM mytable")) 

Read More

PHP: Warning mcrypt_generic_init(): Iv size is incorrect; supplied length: 12, needed: 8

Basic Facts:

$algorithm  = MCRYPT_BLOWFISH; $mode       = MCRYPT_MODE_CBC; $randSource = MCRYPT_DEV_URANDOM; 

Note This is not a strict coding question.

Context:

CentOS 7, Apache 2.4.12, & PHP 5.6.20.

I am making an HTML email with a "verify your email address" link that allows a registration process to complete. Everything on my virtual private server is UTF-8, and all form and query string input is processed with multi-byte (mb) funcions.

Background

As an experiment (I know about the age and state of the mcrypt library), I am attempting to decrypt Blowfish encrypted query string parameters. Assume that on the way up, the encryption sequence works perfectly and I am receiving email with the link.

On the way down, the hmac_hash() signing (SHA-512, just for this experiment) is working and I am able to separate each independent message (32 characters) from its hash checksum (128 characters). Base64 decoding of the separated message portion is working. For each parameter, I am left with the composite cipher text, where composite cipher text equals the IV + base cipher text. Assume I use a version of substr() to obtain the IV and the base cipher text independently (which is par for the course).

Problem

PHP: Warning  mcrypt_generic_init(): Iv size is incorrect; supplied length: 12, needed: 8 

Assume I have combed the PHP manual and Stackoverflow. Assume I have looked at other questions similar, but not exactly like this one. Assume I have searched the Internet to no avail. Assume I have enough experience to setup mb_string properly. Assume that I will take care of mcrypt padding when I get past this current problem.

Could multi-byte issues be interfering with decryption?

Could base64 encoding the IV + base cipher text corrupt the IV?

Could base64 padding be an issue?

Should I be specifying a more specific MCRYPT_BLOWFISH_*?

Why does the blowfish IV size report 8 bytes, but rarely produces an 8 byte IV?

Which substr() should I use, substr() or mb_substr(), for a setup that leans towards making everything UTF-8 and processes all other input as multi-byte UTF-8. I know that is an odd question, but all of the PHP Manual mycrypt decryption sequence examples use substr(), and none use mb_substr(). Everything on my site works with mb_functions when possible, and I would not mind using substr() if it solved my problem, but it does not solve it. When I use mb_substr(), I get the following warning.

PHP: Warning  mcrypt_generic_init(): Iv size is incorrect; supplied length: 11, needed: 8 

Does anyone have any experience with this exact issue? Constructive answers will be rewarded!

Latest

Above is an example Blowfish hash that I am trying to reconstruct from an array, received via a SHA512 HMACed, symmetricly Blowfish encrypted (CBC), url safe Base64 encoded, urlencoded, query string (phew!).

Below, this is what the strings for the query string look like before being urlencoded (don't mind the indexes right now).

Above is the Base64 decoded and Blowfish decrypted array derived from the query string (Obviously, there are security steps in between this result, but I am just trying to show the latest state of things.) Something is not right. Encryption appears to work without any errors. Decryption does not produce and errors either. The plain text is just wrong. If I join/implode these elements, they will not be like the blowfish hash above.

1 Answers

Answers 1

Preliminary answer.

Both mycrpt_create_iv() and mcrypt_generic() output an ISO-8859-1 string. If you have a pure UTF-8 stack, and you intend on using Mcrypt's Blowfish to encrypt query string values, you must ensure that the composite cipher text (IV + Cipher Text) arrives at the decryption sequence as ISO-8859-1 (so, to be safe, you better make sure it arrives at the encryption sequences as ISO-8859-1). You can use substr() to recover the IV and true ciper text just fine, but you must first use utf8_decode() to change your UTF-8 composite cipher text to single byte ISO-8859-1 (testing in progress).

In effect, it seems that you want to be using ISO-8859-1 with mcrypt. Shape your inputs accordingly. But, still, now that the IV error is gone the data does not decrypt.

Read More

Cakephp response cannot read UTF-8 file name

I want to download the file after login check so wrote a function in my controller like

// Function to check login and download News PDF file public function download(){      if($this->Auth->user()){          // Get the news file path from newsId          $pNewsObj  = ClassRegistry::init('PublicNews');         $news = $pNewsObj->findById($newsId);          $filePath = ROOT.DS.APP_DIR.DS.'webroot/upload_news'.DS.$news['PublicNews']['reference'];         // Check if file exists         if(!file_exists($filePath)){             return $this->redirect('/404/index.php');         }         $this->response->charset('UTF-8');         //$this->response->type('pdf');         $this->response->file('webroot/upload_news'.DS.$news['PublicNews']['reference'],  array('download' => true, 'name' => $news['PublicNews']['reference']));         //$this->response->download($news['PublicNews']['reference']);         return $this->response;     }else{         return $this->redirect(array('controller'=> 'users', 'action' => 'login'));     } }  

Now, everything works fine as required.

PROBLEM : when the file name is in UTF-8 eg. テスト.pdf (its Test.pdf in japanese) cakephp throws error like this.

For English filename it works perfectly fine but my client wants the filename should be the same as uploaded, so I can't change the filename to English.

3 Answers

Answers 1

If you want to know character encoding, you can use mb_detect_encoding() function if input text has enough length to detect encoding. But I am guessing your client would upload SJIS file. Because most Japanese people are using SJIS, as Windows has adopted SJIS for Japanese language.

I confirmed your code in my local environment. As cake's File class seems to be not able to handle SJIS correctly, you cannot use Response::file(). So I wrote alternative code.

public function download(){      if($this->Auth->user()){          // Get the news file path from newsId          $pNewsObj  = ClassRegistry::init('PublicNews');         $news = $pNewsObj->findById($newsId);          if (!$news) {             throw new NotFoundException();         }          $fileName = mb_convert_encoding($news['PublicNews']['reference'], 'SJIS-win', 'UTF8');          // Directory traversal protection         if (strpos($fileName, '..') !== false) {             throw new ForbiddenException();         }          $filePath = WWW_ROOT . 'upload_news' . DS . $fileName;         if (!is_readable($filePath)) {             throw new NotFoundException();         }          if (function_exists('mime_content_type')) {             $type = mime_content_type($filePath);             $this->response->type( $type );         } else {             // TODO: If Finfo extension is not loaded, you need to detect content type here;         }          $this->response->download( $fileName );         $this->response->body( file_get_contents($filePath) );          return $this->response;     }else{         return $this->redirect(array('controller'=> 'users', 'action' => 'login'));     } }  

However, I recommend you to convert SJIS to UTF8 before save it into your database and your disk. It is difficult to handle SJIS characters without enough knowledge about it. Because SJIS characters may contain ascii characters in the second byte. Especially backslash (\) is most dangerous. For example, 表 (955C) contains a backslash (5C = backslash). Note that I am not talking about rare cases. 表 means table or appearance in Japanese. 十 also contains a backslash and it means 10 in Japanese. 能 also contains a backslash and it means skill.

Unlike UTF-8 byte sequence, if you handle SJIS characters, almost all string functions don't work correctly. explode() would break SJIS byte sequence. strpos() would return wrong result. Does your client connect to your server by using FTP or SCP directly? If not, it would be better to convert SJIS to UTF-8 before save, and re-convert UTF-8 to SJIS before return to your client.

Answers 2

If you like you can change the file name before uploading the file so at time of downloading this error will not happen.

      public function change_file_name($fileName= '') {         $ext            =   pathinfo($fileName, PATHINFO_EXTENSION);         $fileName       =   'file_'.time().".".$ext;         $exFileName     =   strtolower(substr($fileName,strrpos($fileName,".") + 1));         $sampleFileName =   str_replace('.'.$exFileName,'', $fileName);         $name           =   Sanitize::paranoid($sampleFileName,array('_'));         $fileRename     =   $name.'.'.$exFileName;         return $fileRename;     }  

Call this function before uploading the file

      $return_file_name   =     $this->change_file_name($file_name);     if($this->moveUploadedFile($tmp_name,WEBSITE_PROFILE_ROOT_PATH.$return_file_name)){         $saveData['profile_image']          =   $return_file_name;     }  

I know this is not proper answer for your case.For this you can make a function like this which will fetch data from database and automatic rename all your save file and update it in your database

Answers 3

Some more information about your client's specifications would help greatly, but Tom Scott found base64 to be the simplest method of making Unicode characters work correctly in PHP.

Depending on how crucial the preservation of filenames in storage is, a solution could be to encode the filenames in base64 when files are uploaded, and reverse the encoding on download. You can then know that you are dealing with ASCII, which should be much more likely to work correctly.

You may need to replace / characters with %2F to make it work.

Hope this helps,
Issa Chanzi

Read More