title: In Perl, why is a utf-8 string printed differently when split into characters?
body: <p>A specially constructed string is printed differently when I use</p>
  
  <pre><code>print $b;
  </code></pre>
  
  <p>or</p>
  
  <pre><code>print for split //, $b;
  </code></pre>
  
  <p>A minimal example is:</p>
  
  <pre><code>#!perl
  use warnings;
  use strict;
  
  use Encode;
  
  my $b = decode 'utf8', "\x{C3}\x{A1}\x{E2}\x{80}\x{93}\x{C3}\x{A1}"; # 'á–á' in Unicode;
  
  print $b, "\n";
  print for split //, $b
  </code></pre>
  
  <p>The output on the console screen (I think I use cp860) is:</p>
  
  <pre><code>Wide character in print at xx.pl line 9.
  ├íÔÇô├í
  Wide character in print at xx.pl line 10.
  ßÔÇôß
  </code></pre>
  
  <p>or in hex:</p>
  
  <pre><code>C3 A1 E2 80 93 C3 A1 
  E1 E2 80 93 E1
  </code></pre>
  
  <p>(separated by <code>0D 0A</code> of course, i.e., <code>\r\n</code>).</p>
  
  <p>The question is WHY is the character rendered differently? </p>
  
  <p>Surprisingly, the effect disappears without the em-dash. The effect is seen for longer strings, as the following example shows.</p>
  
  <p>For the string 'Él es mi tío Toño –Antonio Pérez' (typed as Unicode in the program; note that the two lines are different!):</p>
  
  <pre><code>Wide character in print at xx.pl line 14.
  ├ël es mi t├¡o To├▒o ÔÇôAntonio P├®rez
  Wide character in print at xx.pl line 15.
  ╔l es mi tÝo To±o ÔÇôAntonio PÚrez
  </code></pre>
  
  <p>However, for the string 'Él es mi tío Toño, Antonio Pérez':</p>
  
  <pre><code>╔l es mi tÝo To±o, Antonio PÚrez
  ╔l es mi tÝo To±o, Antonio PÚrez
  </code></pre>
  
  <p>nothing bad happens, and the two lines are rendered in the same way. The only difference is the presence of an en-dash <code>–</code>, i.e., <code>'\x{E2}\x{80}\x{93}'</code>!</p>
  
  <p>Also, <code>print join '', split //, $b;</code> gives the same result as <code>print $b;</code> but different from <code>print for split //, $b;</code>.</p>
  
  <p>If I add <code>binmode STDOUT, 'utf8';</code>, then both outputs are <code>ÔÇô├í</code> = E2 80 93 C3 A1.</p>
  
  <p>So my question is not exactly about how to avoid it, but about why this happens: why does the same string behave differently when split? </p>
  
  <p>Apparently in both cases the <code>utf8</code> flag is on. Here is a more detailed program that shows more information about both strings: <code>$a</code> before <code>decode</code> and <code>$b</code> after <code>decode</code>:</p>
  
  <pre><code>#!perl
  use warnings;
  use strict;
  use 5.010;
  
  use Encode;
  
  my $a = "\x{C3}\x{A1}\x{E2}\x{80}\x{93}\x{C3}\x{A1}"; # 'á–á' in Unicode;
  my $b = decode 'utf8', $a;
  
  say '------- length and utf8 ---------';
  say "Length (a)=", length $a, ", is_uft8(a)=", (Encode::is_utf8 ($a) // 'no'), ".";
  say "Length (b)=", length $b, ", is_uft8(b)=", (Encode::is_utf8 ($b) // 'no'), ".";
  say '------- as a variable---------';
  say "a: $a";
  say "b: $b", ' &lt;== *** WHY?! ***';
  say '------- split ---------';
  print "a: "; print for split //, $a; say '';
  print "b: "; print for split //, $b; say ' &lt;== *** DIFFERENT! ***';
  say '------- split with spaces ---------';
  print "a: "; print "[$_] " for split //, $a; say '';
  print "b: "; print "[$_] " for split //, $b; say '';
  say '------- split with properties ---------';
  print "a: "; print "[$_ is_utf=" . Encode::is_utf8 ($_) . " length=" . length ($_) . "] " for split //, $a; say '';
  print "b: "; print "[$_ is_utf=" . Encode::is_utf8 ($_) . " length=" . length ($_) . "] " for split //, $b; say '';
  say '------- ord() ---------';
  print "a: "; print ord, " " for split //, $a; say '';
  print "b: "; print ord, " " for split //, $b; say '';
  </code></pre>
  
  <p>and here is its output on the console:</p>
  
  <pre><code>------- length and utf8 ---------
  Length (a)=7, is_uft8(a)=.
  Length (b)=3, is_uft8(b)=1.
  ------- as a variable---------
  a: ├íÔÇô├í
  Wide character in say at x.pl line 16.
  b: ├íÔÇô├í &lt;== *** WHY?! ***
  ------- split ---------
  a: ├íÔÇô├í
  Wide character in print at x.pl line 19.
  b: ßÔÇôß &lt;== *** DIFFERENT! ***
  ------- split with spaces ---------
  a: [├] [í] [Ô] [Ç] [ô] [├] [í]
  Wide character in print at x.pl line 22.
  b: [ß] [ÔÇô] [ß]
  ------- split with properties ---------
  a: [├ is_utf= length=1] [í is_utf= length=1] [Ô is_utf= length=1] [Ç is_utf= length=1] [ô is_utf= length=1] [├ is_utf= length=1] [í is_utf= length=1]
  Wide character in print at x.pl line 25.
  b: [ß is_utf=1 length=1] [ÔÇô is_utf=1 length=1] [ß is_utf=1 length=1]
  ------- ord() ---------
  a: 195 161 226 128 147 195 161
  b: 225 8211 225
  </code></pre>