All Projects FreeCAD  Addon Manager  Arch  Assembly  Draft  Expressions  FEM  File formats  GCS  OpenSCAD  Part  PartDesign  Path  Project Tools & Websites  Raytracing  Robot  Sketcher  Spreadsheet  TechDraw

View Issue Details

Related ChangesetsJump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0001027FreeCADBugpublic2013-02-25 09:392015-01-24 22:46
Reportershoogen Assigned Toshoogen  
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionfixed 
Product Version0.12 
Fixed in Version0.15 
Summary0001027: UTF-8 encoded PyString objects are used instead of unicode objects to pass unicode data to the python interpreter
DescriptionUnicode Data should be converted to Python objects using

PyObject* PyUnicode_DecodeUTF8(const char *s, Py_ssize_t size, const char *errors)
or
PyObject* PyUnicode_DecodeUTF8Stateful(const char *s, Py_ssize_t size, const char *errors, Py_ssize_t *consumed)

A proper Unicode aware implementation is used for the FreeCADGui.Console.
In but in most cases FreeCAD encodes unicode data using utf-8 and passes it to the python interpreter as 8bit strings instead of using a proper Unicode object.
This forces the user to handle those strings manualy instead of writing code transparent to unicode objects and to strings that are properly encoded.

For example. The workaround to handle utf-8 encoded file names on an not utf-8 based filesystem. Will break properly encoded strings using the filesystemencoding. (https://sourceforge.net/apps/phpbb/free-cad/viewtopic.php?f=8&t=3589&p=28576)
Tagsunicode
FreeCAD Information

Relationships

related to 0001731 closedshoogen FreeCAD spreadsheet csv import/export - path with umlauts 
related to 0001747 closedshoogen FreeCAD QString::toLocal8Bit() strips non-latin charaters 
related to 0001763 closedshoogen FreeCAD use UTF8 for filenames 
parent of 0001744 closedwmayer FreeCAD Use unicode in Unit system 
parent of 0001852 closedshoogen FreeCAD The python interpreter does not recognize the posix locale settings 
related to 0001411 closedshoogen FreeCAD Importing and Exporting fails on Windows due to filepath encoding 
related to 0001795 closedyorik Arch adapt IFC exporters to accept unicode filenames 

Activities

shoogen

2014-07-11 19:37

developer   ~0004787

also present in unit system
http://forum.freecadweb.org/viewtopic.php?f=23&t=6923&p=56527#p56527

shoogen

2014-09-06 09:27

developer   ~0005077

http://stackoverflow.com/questions/10180765/open-file-with-a-unicode-filename

shoogen

2014-09-08 07:50

developer   ~0005079

Currently we use
doCommand(Doc,"f.FileName = \"%s\"",(const char*)fn.toUtf8());
the simplest alternative would be to change it to
doCommand(Doc,"f.FileName = unicode(\"%s\",\"utf-8\")",(const char*)fn.toUtf8());
This will leave a characters readable in the console.

Another solution would be to create suitable escaped unicode literal strings.

This will fix the problems for all python modules. However, the C++ modules would need to be adapted to handle.
I would suggest to change the c++ modules like the python open function.
If the parameter is a string, it is expected to encoded correctly.
If the parameter is a unicode string, it encoded by the by the function.
Therefore i would change a the import and open functions to save the filename paramter to a PyObject. Then we would add a function to Base::Interpreter to convert it to a 8bit cstring.
For a unicode object this would mean to encode the string with the encoding returned by sys.getfilesystemencoding().
On most linux system this would be utf-8. On windows this would be 'mbcs'.

shoogen

2014-12-16 13:35

developer   ~0005392

needs workaround for the 0.15 release.

shoogen

2015-01-24 13:18

developer   ~0005704

we need to address
src/Gui/WidgetFactory.cpp:431
after a fixing 0001852

Related Changesets

FreeCAD: master a53a239f

2014-09-21 23:45:32

shoogen


Committer: wmayer Details Diff 		issue 0001027 use unicode filepaths Affected Issues
0001027
mod - src/App/Application.cpp Diff File
mod - src/App/ApplicationPy.cpp Diff File
mod - src/Base/Tools.cpp Diff File
mod - src/Base/Tools.h Diff File
mod - src/Gui/Application.cpp Diff File
mod - src/Gui/ApplicationPy.cpp Diff File
mod - src/Mod/Cam/App/AppCamPy.cpp Diff File
mod - src/Mod/Drawing/Gui/AppDrawingGuiPy.cpp Diff File
mod - src/Mod/Drawing/Gui/Command.cpp Diff File
mod - src/Mod/Fem/App/AppFemPy.cpp Diff File
mod - src/Mod/Fem/App/FemMeshPyImp.cpp Diff File
mod - src/Mod/Image/Gui/AppImageGuiPy.cpp Diff File
mod - src/Mod/Image/Gui/Command.cpp Diff File
mod - src/Mod/Import/App/AppImportPy.cpp Diff File
mod - src/Mod/Import/App/CMakeLists.txt Diff File
mod - src/Mod/Import/Gui/AppImportGuiPy.cpp Diff File
mod - src/Mod/JtReader/App/AppJtReaderPy.cpp Diff File
mod - src/Mod/Mesh/App/AppMeshPy.cpp Diff File
mod - src/Mod/Part/App/AppPartPy.cpp Diff File
mod - src/Mod/Part/App/CMakeLists.txt Diff File
mod - src/Mod/Part/App/ImportStep.cpp Diff File
mod - src/Mod/Part/App/TopoShape.cpp Diff File
mod - src/Mod/Part/App/TopoShapePyImp.cpp Diff File
add - src/Mod/Part/App/encodeFilename.h Diff File
mod - src/Mod/Points/App/AppPointsPy.cpp Diff File
mod - src/Mod/Raytracing/Gui/AppRaytracingGuiPy.cpp Diff File
mod - src/Mod/Sketcher/App/AppSketcherPy.cpp Diff File
mod - src/Mod/Sketcher/Gui/AppSketcherGuiPy.cpp Diff File

Issue History

Date Modified Username Field Change
2013-02-25 09:39 shoogen New Issue
2013-04-23 22:05 yorik Relationship added parent of 0000995
2014-07-11 19:37 shoogen Note Added: 0004787
2014-09-04 15:46 shoogen Relationship added related to 0001731
2014-09-06 09:27 shoogen Note Added: 0005077
2014-09-08 07:50 shoogen Note Added: 0005079
2014-09-13 20:22 shoogen Relationship added parent of 0001744
2014-09-15 17:22 shoogen Tag Attached: unicode
2014-09-22 08:30 shoogen Relationship added related to 0001747
2014-09-22 09:48 shoogen Assigned To => shoogen
2014-09-22 09:48 shoogen Status new => assigned
2014-09-24 11:32 shoogen Relationship added related to 0001411
2014-09-24 11:41 shoogen Relationship added related to 0001763
2014-10-11 12:30 wmayer Changeset attached => FreeCAD Master master a53a239f
2014-10-11 12:43 shoogen Relationship deleted parent of 0000995
2014-10-11 12:44 shoogen Status assigned => closed
2014-10-11 12:44 shoogen Resolution open => fixed
2014-10-22 07:32 shoogen Relationship added related to 0001795
2014-12-16 13:34 shoogen Relationship added parent of 0001852
2014-12-16 13:35 shoogen Note Added: 0005392
2014-12-16 13:35 shoogen Status closed => feedback
2014-12-16 13:35 shoogen Resolution fixed => reopened
2015-01-24 13:18 shoogen Note Added: 0005704
2015-01-24 13:18 shoogen Status feedback => assigned
2015-01-24 22:46 shoogen Status assigned => closed
2015-01-24 22:46 shoogen Resolution reopened => fixed
2015-01-24 22:46 shoogen Fixed in Version => 0.15