FreeCADView Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0001852 | FreeCAD | Bug | public | 2014-12-08 13:46 | 2017-02-10 10:50 |
| Reporter | shoogen | Assigned To | shoogen | ||
| Priority | high | Severity | major | Reproducibility | always |
| Status | closed | Resolution | fixed | ||
| OS | POSIX | ||||
| Product Version | 0.15 | ||||
| Target Version | 0.15 | ||||
| Summary | 0001852: The python interpreter does not recognize the posix locale settings | ||||
| Description | This currently breaks the usage of non-ascii characters in filenames for import and export handled in python modules. | ||||
| Steps To Reproduce | $ python -c "import sys;print sys.getfilesystemencoding()" UTF-8 $ LANG=C python -c "import sys;print sys.getfilesystemencoding()" ANSI_X3.4-1968 $ ./fc-build/bin/FreeCADCmd [FreeCAD Console mode <Use Ctrl-D (i.e. EOF) to exit.>] >>> import sys >>> sys.getfilesystemencoding() 'ANSI_X3.4-1968' >>> import locale >>> locale.getdefaultlocale() (None, None) | ||||
| Tags | encoding, locale, sys.getfilesystemencoding, unicode | ||||
| FreeCAD Information | |||||
| related to | 0002891 | closed | wmayer | Sketcher | Sketching impossible, Type.Error Exception |
| related to | 0001545 | closed | FreeCAD | Folders with tick in the name cannot be opened | |
| related to | 0002018 | closed | wmayer | FreeCAD | Running Freecad with specific locale causes freeze |
| child of | 0001027 | closed | shoogen | FreeCAD | UTF-8 encoded PyString objects are used instead of unicode objects to pass unicode data to the python interpreter |
| child of | 0001930 | closed | shoogen | Drawing | Export a page to an SVG file and UnicodeEncodeError |
|
|
http://www.mevislab.de/docs/current/MeVis/ThirdParty/Documentation/Publish/Python/library/locale.html > When Python code uses the locale module to change the locale, this also affects the embedding application. If the embedding application doesn’t want this to happen, it should remove the _locale extension module (which does all the work) from the table of built-in modules in the config.c file, and make sure that the _locale module is not accessible as a shared library. Great idea python ;) |
|
|
https://github.com/5263/FreeCAD/commit/c85ddd166ef80e747719549922bb7e5ddffc58cd https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=559846 |
|
|
Werner, could you please comment on the following patch https://github.com/5263/FreeCAD/compare/dev-locale diff --git a/src/Main/MainCmd.cpp b/src/Main/MainCmd.cpp index 17cb6c1..b7c1d68 100644 --- a/src/Main/MainCmd.cpp +++ b/src/Main/MainCmd.cpp @@ -70,4 +70 @@ int main( int argc, char ** argv ) -#if defined(FC_OS_LINUX) - putenv("LANG=C"); - putenv("LC_ALL=C"); -#else + setlocale(LC_ALL, ""); @@ -75 +71,0 @@ int main( int argc, char ** argv ) -#endif diff --git a/src/Main/MainGui.cpp b/src/Main/MainGui.cpp index 300084a..a0050c7 100644 --- a/src/Main/MainGui.cpp +++ b/src/Main/MainGui.cpp @@ -194,2 +194 @@ int main( int argc, char ** argv ) - putenv("LANG=C"); - putenv("LC_ALL=C"); + putenv("LC_NUMERIC=C"); @@ -199,2 +198 @@ int main( int argc, char ** argv ) - putenv("LANG=C"); - putenv("LC_ALL=C"); + putenv("LC_NUMERIC=C"); |
|
|
It seems OCCT fixed the problem in version 6.6.0 http://tracker.dev.opencascade.org/view.php?id=22898 |
|
|
I can confirm that "putenv("LC_NUMERIC=C");" is sufficient now to fix the OCC problem -- tested with OCC 6.7.1 |
|
|
I see two possibilities. 1. Don't touch the locales before the 0.15 release this will require a different workaround for different platforms. 2. Do tackle this problem before the 0.15 release. this requires testing on all platforms before the release. Thank you Werner for testing. I don't understand why to use putenv (instead of setlocale) on linux. It would make sense on windows, because windows has no fork command. If putenv is nessesary because of a setlocale used somewhere is (in QT for example) and if other parts of FreeCAD rely on LC_NUMERIC=C (like the xml writer and parser) then, my patch would be a bad idea. However, the xml output seems to unaffected. |
|
|
I go for possibility 1) as it's too dangerous to do this change for an imminent release. Thinking further about why putenv works but setlocale fails I think putenv affects the locale settings for C and C++ while setlocale affects it only for C. The C++ way to change the locale is to do: std::locale::global(std::locale("C")) but it should be checked if there is a way to change only numeric input/output > However, the xml output seems to unaffected. Here we explicitly change the locale settings of a concrete stream instance. Search for "imbue" in the sources. |
|
|
Maybe just writing std::locale::global(std::locale("C", std::locale::numeric)); does the trick. See also: http://stackoverflow.com/questions/12373341/does-stdlocaleglobal-make-affect-to-printf-function |
|
|
git://github.com/5263/FreeCAD dev-locale-cpp std::locale::global(std::locale(std::locale(""),std::locale("C"),std::locale::numeric)); seems to set the locale correctly (on linux). On my system it has the "name" LC_CTYPE=de_DE.UTF-8;LC_NUMERIC=C;LC_TIME=de_DE.UTF-8;LC_COLLATE=de_DE.UTF-8;LC_MONETARY=de_DE.UTF-8;LC_MESSAGES=de_DE.UTF-8;LC_PAPER=de_DE.UTF-8;LC_NAME=de_DE.UTF-8;LC_ADDRESS=de_DE.UTF-8;LC_TELEPHONE=de_DE.UTF-8;LC_MEASUREMENT=de_DE.UTF-8;LC_IDENTIFICATION=de_DE.UTF-8 and having a name allows to set the C locale as well. |
|
|
$ git grep -l strtod src/3rdParty/CxImage/png/CHANGES src/3rdParty/CxImage/png/pngrutil.c src/3rdParty/salomesmesh/src/SMESH/SMESH_Pattern.cpp $ git grep -l atof src/3rdParty/CxImage/png/libpng-1.2.24.txt src/3rdParty/salomesmesh/inc/UNV_Utilities.hxx src/Base/Parameter.cpp src/Base/Quantity.cpp src/Base/Reader.cpp src/Mod/Cam/App/ConvertDyna.cpp src/Mod/Fem/App/AppFemPy.cpp src/Mod/Fem/App/FemMesh.cpp src/Mod/Mesh/App/Core/MeshIO.cpp src/Mod/Points/App/PointsAlgos.cpp src/Mod/Robot/App/Robot6Axis.cpp $ git grep -l imbue src/Base/Reader.cpp src/Base/Writer.cpp > Search for "imbue" in the sources. Hm, I'm not amused. |
|
|
with the latest change i can't open exiting documents. ;( |
|
|
Qt ;( |
|
|
Which change? |
|
|
mine git://github.com/5263/FreeCAD dev-locale-cpp http://www.qtcentre.org/threads/53127-Override-locale-with-QSystemLocale |
|
|
Ok, now i understand the need for the putenv on unixoid OSes. The damn Qt uses qgetenv to setup the locale. https://qt.gitorious.org/qt/qtbase/source/e0a8b5ce88bc50440dcec2fe3a86d83e2a7dc7b0:src/corelib/tools/qlocale_unix.cpp#L69 |
|
|
If you don't want to modify the locale we have to use old method of passing utf8 byte strings in the 0.15 release (at least for unixoid platforms) git://github.com/5263/FreeCAD ugly_workaround_for_bug_1852 |
|
|
I haven't checked yet the branch dev-locale-cpp. Does this fix the issue reliably? When I said that I would postpone the fix for 0.16 then I meant to leave bug in for 0.15. But since you want to have a solution already for 0.15 then I think it's better to apply the proper solution instead of reverting any commits because this needs a lot of testing anyway and is as your branch name says really "ugly". |
|
|
I updated dev-locale-cpp I looked promissing for normal spinboxes. But I got problems with quantity spinboxes. These persisted when i did an icremental build with the version of the master branch. (I'm waiting for a clean build of master before i continue testing) And finally I have to use the proper namespace in c++ ;) |
|
|
Write this: class AppExport DecimalSystemLocale : public QSystemLocale IMO, in case this class is needed it would make more sense to move this to "Base". In this case the export macro must be "BaseExport". |
|
|
To me it seems that the locale works in QT, but now the Quantity Parser is broken. |
|
|
One thing I still do not understand is why something for Qt needs to be fixed. The story is: 1. All the trouble started because OCC doesn't handle numeric input properly. 2. That's why setting LANG and LC_ALL to C solved this issue. 3. Since this affects Qt "QLocale::system();" is called very first to set up the locale stuff for Qt and QFile::setEncodingFunction/QFile::setDecodingFunction is used to utf-8 handling 4. However, setting LC_ALL and LANG affects Python and since OCC now improved the handling for numeric input setting LC_NUMERIC seems to be sufficient. Now, does changing LC_NUMERIC affect Qt in a certain way? Btw, for MacOSX nothing needs to be done because with current master "import sys;print sys.getfilesystemencoding()" returns UTF-8 and applying the dev-locale-cpp branch makes the applicationcrash immediately. |
|
|
Qt has the ability to set the deflection Property to 0 when loading an existing FreeCAD file with an unexpected decimal seperator. And OCCT starts mesheshing with 0 deflection. ;) 1. That should be fixed by 6.6.0 2. That did not prevent Jürgen from using atof in the QuantityParser. 3. That made sense for Qt < 4.2. But I would consider this as not beautiful. And using a different locale in Qt, c++ and the python interpreter is messy. When I'm touching the code anyway I try, to make it more beautiful. If that's not possible we have to live with that. But Qt allows to set the decimal seperator independenly. 4. I agree for Linux. But I don't think Qt will respect it on windows. |
|
|
It seems that std::locale::global(std::locale(std::locale(""),std::locale("C"), std::locale::numeric)); is not enough for the atof() in src/Base/Quantity.cpp to accept a '.' as decimal seperator on linux with my locale |
|
|
neither is a std::setlocale() |
|
|
> Qt has the ability to set the deflection Property to 0 when loading an existing FreeCAD file with an unexpected decimal seperator. How do you get a FreeCAD project with an unexpected decimal separator (i.e. another one than ".")? > That did not prevent Jürgen from using atof in the QuantityParser. If this causes any trouble we can easily use C++ streams instead. There you can set on a single instance (not globally) use the C locale. > And using a different locale in Qt, c++ and the python interpreter is messy. I disagree. For low-level handling (e.g file reading, writing, ...) I would like to use a locale that is platform independent, i.e. the "C" locale. For user input in the GUI (i.e. Qt) I would like to use the system locale because it would be extremely annoying when in another application I use the system locale and suddenly in FreeCAD it would be the C locale. |
|
|
I haven't so far i only opened documents when qt expected a comma. Yes, i have done it for the quantity parser in my branch. But atof is also used in third party libs like CxImage. I partly agree. It would make sense if we devide the functionalty beetween frontend and backend. But we can't draw the line between c, c++, qt and python. We use qstrings in the properties handling and python to interface fileformats and gui. > For user input in the GUI (i.e. Qt) I would like to use the system locale because it would be extremely annoying when in another application I use the system locale and suddenly in FreeCAD it would be the C locale. Thats a good idea. But this means that you would need to use a locale object when handling properties. |
|
|
Ok, its been an interesting but disappointing excursus. Lets just use putenv in the manner it is currently used, but for LC_NUMMERIC. Thus making the minimal possible change. I filed a pull-request. (without the unit test, because it won't test the behavior correctly on windows) |
|
|
This explains all the trouble we had related to locales: https://forum.freecadweb.org/viewtopic.php?f=3&t=20537#p158962 So, it's caused by Qt by calling setlocale(LC_ALL,"") which then applies the system-wide locale settings. So, on systems which set LC_ALL (e.g. Debian) this leads to problems especially if the decimal separator is a comma. I played a bit with that and made the observation that the decimal separator of floating point numbers in the user.cfg will become a comma and then it can happen that the next time it cannot handle the conversion from text to float and sets it to 0.0. Especially for the tessellation of shapes this leads to massive problems because then the deviation is 0.0 instead of a sensible value and causes the tessellation of curved shapes to take forever. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2014-12-08 13:46 | shoogen | New Issue | |
| 2014-12-08 13:48 | shoogen | Steps to Reproduce Updated | |
| 2014-12-08 13:49 | shoogen | Steps to Reproduce Updated | |
| 2014-12-08 14:01 | shoogen | Note Added: 0005360 | |
| 2014-12-08 14:02 | shoogen | Note Edited: 0005360 | |
| 2014-12-08 14:04 | shoogen | OS | Linux => POSIX |
| 2014-12-08 14:04 | shoogen | Summary | The python interpreter does not recognize the locale settings => The python interpreter does not recognize the posix locale settings |
| 2014-12-08 14:22 | shoogen | Tag Attached: sys.getfilesystemencoding | |
| 2014-12-08 14:22 | shoogen | Tag Attached: unicode | |
| 2014-12-08 14:23 | shoogen | Tag Attached: encoding | |
| 2014-12-09 16:21 | shoogen | Note Added: 0005361 | |
| 2014-12-09 16:22 | shoogen | Note Edited: 0005361 | |
| 2014-12-09 16:37 | shoogen | Note Added: 0005362 | |
| 2014-12-09 16:37 | shoogen | Assigned To | => wmayer |
| 2014-12-09 16:37 | shoogen | Status | new => assigned |
| 2014-12-09 19:38 | shoogen | Note Added: 0005363 | |
| 2014-12-09 19:38 | shoogen | Note Edited: 0005363 | |
| 2014-12-11 10:54 | wmayer | Note Added: 0005365 | |
| 2014-12-15 10:03 | shoogen | Note Added: 0005385 | |
| 2014-12-15 10:48 | wmayer | Note Added: 0005386 | |
| 2014-12-15 11:05 | wmayer | Note Added: 0005387 | |
| 2014-12-16 13:34 | shoogen | Relationship added | child of 0001027 |
| 2014-12-16 19:04 | shoogen | Note Added: 0005396 | |
| 2014-12-16 19:04 | shoogen | Note Edited: 0005396 | |
| 2014-12-16 21:05 | shoogen | Note Edited: 0005396 | |
| 2014-12-16 21:08 | shoogen | Note Added: 0005399 | |
| 2014-12-17 18:17 | shoogen | Note Added: 0005401 | |
| 2014-12-17 19:13 | shoogen | Note Added: 0005402 | |
| 2014-12-17 19:31 | shoogen | Note Edited: 0005402 | |
| 2014-12-17 19:41 | wmayer | Note Added: 0005403 | |
| 2014-12-17 19:51 | shoogen | Note Added: 0005404 | |
| 2014-12-18 07:00 | shoogen | Note Added: 0005405 | |
| 2014-12-21 12:08 | shoogen | Note Added: 0005414 | |
| 2014-12-21 12:28 | wmayer | Note Added: 0005415 | |
| 2014-12-21 16:52 | shoogen | Note Added: 0005420 | |
| 2014-12-21 17:17 | wmayer | Note Added: 0005422 | |
| 2014-12-21 17:44 | shoogen | Note Added: 0005423 | |
| 2014-12-21 18:19 | wmayer | Note Added: 0005424 | |
| 2014-12-21 18:50 | shoogen | Note Added: 0005425 | |
| 2014-12-21 19:07 | shoogen | Note Added: 0005426 | |
| 2014-12-21 20:07 | shoogen | Note Added: 0005428 | |
| 2014-12-21 20:49 | wmayer | Note Added: 0005429 | |
| 2014-12-21 23:01 | shoogen | Note Added: 0005430 | |
| 2015-01-07 11:38 | wmayer | Relationship added | related to 0001545 |
| 2015-01-24 10:17 | shoogen | Relationship added | child of 0001930 |
| 2015-01-24 18:32 | shoogen | Note Added: 0005714 | |
| 2015-01-24 19:01 | shoogen | Note Edited: 0005714 | |
| 2015-01-24 22:39 | shoogen | Changeset attached | => FreeCAD Master master f32caef4 |
| 2015-01-24 22:39 | shoogen | Assigned To | wmayer => shoogen |
| 2015-01-24 22:39 | shoogen | Status | assigned => closed |
| 2015-01-24 22:39 | shoogen | Resolution | open => fixed |
| 2015-03-21 23:50 | shoogen | Relationship added | related to 0002018 |
| 2017-02-10 10:04 | wmayer | Relationship added | related to 0002891 |
| 2017-02-10 10:18 | wmayer | Note Added: 0008261 | |
| 2017-02-10 10:50 | Kunda1 | Tag Attached: locale |
