HelpSpot Help Desk Software | HelpSpot Blog | HelpSpot Support

Non-latin character sets


#1

Thanks for your help with my earlier queries.

Our team deals with a number of japanese companies, to the extent where we receive issues and send out responses in japanese. These emails are just appearing as ??? within helpspot.

My questions are:

  • are non-latin character sets supported?
  • do i need to set the character set of mysql database differently at all?

I can provide some sample emails if appropriate.

Thanks,


#2

Hi Miles,

You could probably run your installation in all Japanese, but it’s not really possible to run it in a mix. Generally non-latin character sets to not work well in PHP applications though there’s some support for Japanese in PHP. A few customers have tried using UTF8 with some success, but it’s not supported and can cause odd issues in the system.

To do it you’d need to have UTF8 tables, enable the MBString override functions in PHP, set the HelpSpot character set to utf8 in the english language file and you may need to add these lines in the english file:

$GLOBALS[‘DB’]->Execute( ‘SET NAMES utf8;’ );
$GLOBALS[‘DB’]->Execute( ‘SET CHARACTER_SET utf8;’ );

Even with this is may not work and you could have great difficulties long term. If it’s only a few emails here and there perhaps they could just handle them in a mail client?

We hope to provide better support for UTF8 in a future release as PHP itself begins to better support it.


#3

Hi Ian and Miles,

We have applied the suggested changes.

Apache and php seem to be working in UTF-8. Database columns where amended to support UTF-8.

Unable ot override the standard string functions with mbstring module ones.
/usr/local/Zend/etc/php.ini
mbstring.func_overload = 7

The login form breaks as soon as it is enabled. I am unable to determine what exactly is wrong with this login form as the PHP code is not accessible. I have set the override to work only with the mail related functions (value = 1). Let’s see what it gives.

Below are the details of changes I made

Thanks,
Vlad

DETAILS:

  1. amend the database columns to support UTF-8

HS_Bayesian_Corpus.sWord varchar(255)
HS_Category.sCategory varchar(200)
HS_Category.sPersonList longtext
HS_Category.sCustomFieldList longtext
HS_Category_ReportingTags.sReportingTag varchar(255)
HS_CustomFields.fieldName varchar(200)
HS_CustomFields.listItems longtext
HS_Errors.sFile varchar(255)
HS_Errors.sDesc varchar(255)
HS_Filters.sFilterName varchar(255)
HS_Filters.tFilterDef longtext
HS_Forums.sForumName varchar(255)
HS_Forums.sDescription varchar(255)
HS_Forums.tModerators longtext
HS_Forums.sName varchar(50)
HS_Forums.sLabel varchar(50)
HS_Forums_Posts.sName varchar(50)
HS_Forums_Posts.tPost longtext
HS_Forums_Topics.sName varchar(50)
HS_Forums_Topics.sTopic varchar(200)
HS_KB_Books.sBookName varchar(255)
HS_KB_Books.tDescription longtext
HS_KB_Books.tEditors longtext
HS_KB_Chapters.sChapterName varchar(255)
HS_KB_Documents.sFilename varchar(255)
HS_KB_Pages.sPageName varchar(255)
HS_KB_Pages.sPage longtext
HS_Mail_Rules.sRuleName varchar(255)
HS_Mail_Rules.tRuleDef longtext
HS_MailBoxes.sReplyName varchar(50)
HS_MailBoxes.tAutoResponse longtext
HS_MailBoxes.tAutoResponse_HTML longtext
HS_MailBoxes_StuckEmails.sEmailFrom varchar(255)
HS_Person.sFname varchar(50)
HS_Person.sLname varchar(100)
HS_Person.sUsername varchar(100)
HS_Person.tSignature longtext
HS_Person.tSignature_HTML longtext
HS_Person.tMobileSignature longtext
HS_Person.tWorkspace longtext
HS_Person_Photos.sDescription varchar(255)
HS_Portal_Bayesian_Corpus.sWord varchar(100)
HS_Reminder.tReminder longtext
HS_Request.sTitle varchar(255)
HS_Request.sFirstName varchar(255)
HS_Request.sLastName varchar(255)
HS_Request_History.tLog longtext
HS_Request_History.tNote longtext
HS_Request_History.tEmailHeaders longtext
HS_Request_Note_Drafts.tNote longtext
HS_Request_Pushed.sPushedTo varchar(255)
HS_Request_Pushed.sReturnedID varchar(255)
HS_Request_Pushed.tComment longtext
HS_Responses.sResponseTitle varchar(255)
HS_Responses.sFolder varchar(255)
HS_Responses.tResponse longtext
HS_Responses.tResponseOptions longtext
HS_Search_Queries.sSearch varchar(255)
HS_Search_Queries.sFromPage varchar(255)
HS_Search_Queries.sSearchType varchar(255)
HS_Sessions2.expireref varchar(255) NULL
HS_Sessions2.sessdata longtext NULL
HS_Settings.tValue longtext
HS_Time_Tracker.tDescription longtext
HS_luSMS.sName varchar(255)
HS_luSMS.sAddress varchar(255)
HS_luStatus.sStatus varchar(255)

  1. amend the application settings:
    /var/www/helpspot/helpspot/lang/english-us.php

//Character set encoding to use throughout HelpSpot. Changes to this may require you to change the collation of your database
define(‘helpspot_charset’,‘UTF-8’);

//Execute the suggested database charset override SQL (as per Ian Landsman’s suggestion)
$GLOBALS[‘DB’]->Execute( ‘SET NAMES utf8;’ );
$GLOBALS[‘DB’]->Execute( ‘SET CHARACTER_SET utf8;’ );

//Tidy’s character set, options are raw, ascii, latin0, latin1, utf8, iso2022, mac, win1252, ibm858, utf16le, utf16be, utf16, big5, shiftjis
define(‘lg_tidycharset’,‘utf8’);

  1. php.ini changes:

/usr/local/Zend/etc/php.ini

mbstring.language = Neutral ; Set default language to Neutral(UTF-8) (default)
mbstring.internal_encoding = UTF-8 ; Set default internal encoding to UTF-8
mbstring.encoding_translation = On ; HTTP input encoding translation is enabled
mbstring.http_input = auto ; Set HTTP input character set dectection to auto
mbstring.http_output = UTF-8 ; Set HTTP output encoding to UTF-8
mbstring.detect_order = auto ; Set default character encoding detection order to auto
mbstring.substitute_character = none ; Do not print invalid characters
default_charset = UTF-8 ; Default character set for auto content type header
#this overloads the standard functions to use the mbstring enhancment
#mbstring.func_overload = 7 (does not work - login form does not work properly)
mbstring.func_overload = 1 #(override email only)


#4

Hi Ian,

Just to summarise - as Vlad mentions, it seems to work fine with unicode being passed through correctly. However, when we enable the “mbstring override”, we are no longer able to log in to the system.

With the override set to mail only, piped-in japanese mails are fine, but when we reply it’s garbled. Are you able to set this up to see why it should stop us logging in?

Japanese support is quite important for us, so please contact me direct if there’s any other way to move this forward with some speed.

Thanks,


#5

Hi,

Sorry if someone is looking into this and I’m being impatient - I wasn’t sure if it had dropped off the radar as we replied on an old thread.

http://www.userscape.com/helpdesk/index.php?pg=forums.posts&id=6687&pc=4

We’ve done some work on getting helpspot to work with UTF-8, but have hit one problem that we can’t get any further with. If you could look at the above thread and advise, that would be great.

Many thanks for your help,

Miles


#6

Hi Miles/Vlad,

Unfortunately PHP’s support of unicode is very bad which is why we do not currently support unicode. I’ll take a look to see why this might be the case, but I cannot guarantee a fix will be possible given the current limitations.


#7

I’m going to look into it. I suspect there will be many more problems than this though. There are places where the system expects single byte characters. These may or may not break things to a point where the system doesn’t work.

One thing is that you were trying to make it work with Japanese correct? Rather than UTF-8 what about setting it to one of the japanese character sets and seeing how that works.


#8

Very nice vlad! On the subject line, saving the english-us.php file in UTF8 might help, though it may also break other things so you might need to be careful there.