HelpSpot Help Desk Software | HelpSpot Blog | HelpSpot Support

Native characters in e-mail wrongly displayed ni HelpSpot


#1

If I send an e-mail to a HelpSpot polled account with accented characters (for example):

Aquest missatge és de prova

then the accented characters arrive corrupt to the HelpSpot Workspace.

The POP3 server in this case is Microsoft Exchange.

Conversely, If I type accented characters, they appear wrong in the e-mail box of the target user. Users would see this:

Aquest missatge és de prova

I suspect that you are encoding things in UTF-8 but not changing the headers for outgoing headers for outgoing mail. To reproduce the ourgoing mail problem you don’t need Exchange, it also happens if you for example SMTP directly to gmail.com.


#2

Thanks Josep. The multilingual aspects of HelpSpot are definitely less tested than other parts. I’m going to attempt to replicate this right now and post back here when I find a fix.


#3

In terms of the inbound email if you want to use a different char set you can change them in english-us.php. For details on how to do this safely you can see the documentation here:
http://www.userscape.com/helpdesk/index.php?pg=kb.page&id=22

Note that the database table will also need to be setup to store the correct charset and the values at the top of the language file will let you modify the display character set.

As for the outgoing, I’m still looking into that but will post here after more follow up. I do believe you are correct though, that an outbound character set is not given.


#4

Josep, I have a fix for the issue. As you said the outbound email did not have the proper charset added.

The fix will be in the next release, but I’m going to email you shortly with 2 files to change out in your installation which should allow the system to correctly send outbound email.


#5

Regarding incoming mails, I suppose that you mean that I should change the following line:

// Character set information sent in HTML head
define(‘lg_charset’,’ ');

to the ANSI Windows code page.

I’m not sure if I follow you with the required changes to the database table, what do you mean exactly?

Note: This would be equivalent to overriding the code page with View->Encoding->Western Europe (Windows) in Internet Explorer. Actually, if I do that in IE with the default setup, I then see the characters right.

I will test this approach combined with the patch files that you just sent, to see if we can get started (I think so, I’ll figure it out in a few minutes).

However, just let me mention that I think that it would make sense in the long run to enforce UTF-8 as the encoding and just make sure at the critical data entry points (such as e-mail parsing) that the encoding is consistently stored in UTF-8, so that the output is consistently displayed in UTF-8.


#6

Thanks for the feedback Josep.

Yes that would be the line to change which as you said would be the same as make the browser interpret it, but if you change the browser setting then I believe it would change it for all sites, not just HelpSpot. Unless there is a site by site setting for that, which I’m not sure about.

All the defaults in HelpSpot are to UTF-8, but there can still be issues with the database. For instance, you can change the character set on the entire database, tables, and columns. So if HelpSpot tries to put UTF-8 data into a non-UTF data column that can also cause corruption. It doesn’t sound like you’re running into that though, so I think we’re OK there.


#7

With the fix you just mailed and changing all the ‘utf-8’ values to ‘iso-8859-1’, I get the right behaviour for Western Europe languages.

I suppose that we will have issues for Eastern Europe or Asian languages (if they attach some native text to documet an issue), but this is a low impact thing as we have workarounds.

However, in the long term, I think that we should be based in utf-8 to avoid artifacts. I suppose that when we think it is safe to move to utf-8, we should run some conversion script for the database.

Talking about artifacts, I found another one when testing your fix:

This is how to reproduce it. The initial status is a HelpSpot 1.03 install, with the outgoing e-mail fix pacthes.

Steps to reproduce:

  1. Copy the following text to a newly created ticket, putting your e-mail as the client’s.

áéíóúàèìòùÁÉÍÓÚÀÈÌÒÙ

  1. Make it public and send the note to the client.

The mail will appear OK (which means that your fix worked well now that the encoding is utf-8) but you will see that the uppercase O with the grave and the acute accent have the wrong characters in the bug history in the Workplace. The rest of the accented characters are OK.

Are you perhaps using a house-made conversion to UTF-8 for the database that is different from the one used for the outgoing mail? This bug is strange because normally either all characters work or all characters fail. Internationalization issues are normally not that selective…


#8

Thanks for taking the time to work through these, it’s extremely helpful.

This one is odd. One of the big downsides of using PHP is that it doesn’t exactly make internationalization easy. HelpSpot doesn’t have any internal custom UTF conversion functions, but the history notes are passed through several PHP functions and I’m thinking this may be an issue with one of those. I’ll report back when I find something out.