Writing Japanese

Category : Non classé

Multiple writing systems

I’m learning Japanese. Among all the things that you have to learn in Japanese is the writing system which is composed of four parts:

  • Hiragana, a set of around 50 characters, specific to Japanese, which represent syllables and are used for grammar elements and simple words
  • Katakana, a second set of around 50 characters, also specific to Japanese, which represent the same syllables, but are used for foreign words or onomatopoeia in manga
  • Kanjis, the Chinese characters that where imported around six hundred years ago. They represent meanings and have multiple pronunciations depending on context. The official list for schools in Japan contains 2250 of those
  • Romaji, as they call it, are our characters, so we just have to learn how they are included in Japanese, because they are used to make something look modern or if it’s a french word, chic (with various results)

All of these are mixed in normal texts, and there’s no white space between words for example, because the transition from one kind of characters to another already makes it readable. Japanese can also be written vertically, starting in the top right of the page, and that’s pretty unusual for westerners.

As Kanjis have many pronunciations, there is a way to help beginners to read them by showing above them (or on the right when writing vertically) the corresponding Hiraganas. This practice is called Furigana in Japanese, and ruby in western typesetting.

Trying to write

When I first encountered all of this, I was lost but at least I could write by hand on paper. Then I searched for ways to type it on computers or smartphones.

For smartphones with Android, it’s quiet easy to install a Japanese keyboard, as you can download virtual keyboards for any language. But that doesn’t mean it’s easy to use ! That’s where the input methods enter. When typing, the input method shows a list of possible words, mostly in Kanjis. As there are few syllables in Japanese, there are many homophones, with vastly different meanings, so this list is not very helpful to me as a beginner… So I will keep using Hiragana for some time I believe.

When I write on a computer, the problem is different. For now I didn’t try input methods, mostly because I have to use Windows at work, and I chose Linux at home. I didn’t want to learn two ways to enter text, and I don’t control my work computer so I can’t install anything. I’ve found a simple site that makes it easy to write Hiragana, Katakana, and possible to write Kanjis: https://www.lexilogos.com/clavier/nihongo.php. It’s not limited to Japanese. You can write using alphabet letters, and they are converted to the corresponding character automatically.

Helping myself

I’ve started to write some short texts with Kanjis, and I wanted to use Furigana to be able to read later even if I forget Kanjis. I’ve seen that it’s possible to add Furigana in Microsoft Word, but it’s a bit cumbersome, because you have to open a dialog for each word.

So far the easiest solution I found to write Kanjis and their Furigana is HTML, which provides a specific element for this use : ruby. If you have a standard editor, you can write HTML on any system, and include Japanese (if the editor supports UTF-8).

You use it this way:<ruby><rb>Kanjis</rb><rt>Furigana</rt></ruby> 

For example “year” is: <ruby><rb>年</rb><rt>ねん</rt></ruby>(“nen” in this case), and should produce the following visual result, with the Furigana above the Kanji:

年ねん

Only the beginning

I still need to learn many things about Japanese but at least I can write what I already know not only on paper !

KUbuntu 18.10 update : strange X11 problem, complex solution

Category : Linux

Upgrade, then reboot…

After the upgrade from 18.04 to 18.10, SDDM failed to start. This happened with kernel 4.18.0-10 and 4.15.0-36 (it was the chance to discover grub-reboot). The machine is a virtual machine, and PCI devices are using virtio-pci driver for example.

I looked into /var/log/Xorg.0.log and found many error messages, the first one being :

(EE) open /dev/dri/card0: No such file or directory
See Xorg log file and crash file attached.

Then I looked at the output of lspci -vvnn and it seems that no kernel driver manages the VGA Controller. According to the following message, the missing kernel driver may be : bochs-drm or virtio-vga
https://www.redhat.com/archives/libvir-list/2015-August/msg00220.html

Asking for help

As usual when something goes wrong and it seems it’s not my fault, I’ve asked for some help. I opened a bug on Launchpad :
https://bugs.launchpad.net/ubuntu/+source/xorg-server/+bug/1802950

Unfortunately that hasn’t been very helpful, mostly because I have never been able to upload a crash report. The standard crash report tool works under X11, which was of course not available. The command line tools didn’t work for me but I didn’t find why. So I left it sleep for some time

A new hope

There was a hint in the developer’s comments that there was some hope to make it work again in the future: Ubuntu could include the missing module in the kernel they provide. It was removed in 2014, and could be back as there is bug 1795857 from October 2018 asking for it. But such sidelines problem doesn’t get much attention, so it’s not included as of January 2019.

Go to the source

Then I thought that it’s only a module, and I could build it myself from the sources of the kernel. I didn’t know if it was easy under Ubuntu, but I already build kernels on Gentoo for 15 years, so I wanted to give it a try.

First I installed the kernel sources package:

 apt-get install linux-source-4.18.0

When I looked at its content I was disappointed, because it’s only a tarball, but it’s easy enough to decompress:

cd /usr/src/linux-source-4.18.0/
tar xjf linux-source-4.18.0.tar.bz2
cd linux-source-4.18.0/

I tried the usual make oldconfig but some packages are necessary to make it work:

apt-get install bison
apt-get install flex
apt-get install libelf-dev

From this point, I could go forward with my idea:

make oldconfig
make menuconfig

In the menus I searched and enabled the module CONFIG_DRM_BOCHS=m
Then I launched the compilation:

make modules

Now you have time to take a break and get coffee, because Ubuntu enables many modules that are compiled by this command. It also takes space: the whole kernel tree with all compilation results and intermediate files amounts to 15 Gb !

At the end of this compilation, I installed manually the only module I’m interested in, because I don’t want to cripple the modules installed from standard packages:

mkdir /lib/modules/4.18.0-12-generic/kernel/drivers/gpu/drm/bochs
cp drivers/gpu/drm/bochs/*.ko /lib/modules/4.18.0-12-generic/kernel/drivers/gpu/drm/bochs
depmod -a

Now is the moment of truth, I try to load this module in the running kernel:

modprobe bochs-drm

It didn’t work, but it was not obvious because modprobe didn’t show any message, so I searched elsewhere and found something in dmesg output:

bochs_drm: version magic '4.18.17 SMP mod_unload ' should be '4.18.0-12-generic SMP mod_unload '

I didn’t know where this was defined nor if I could do something about it. If it’s not the same version, maybe I’ve the wrong kernel sources !

However I found it in the main Makefile of the kernel so I decided to make a further attempt by changing the values in that file, even though I’ve no idea of the potential consequences… So I changed the values:

SUBLEVEL = 17
EXTRAVERSION =

To the values expected by the running kernel:

SUBLEVEL = 0
EXTRAVERSION = -12-generic

Then I restarted a clean compilation:

make clean
make modules
# Get more coffee
cp drivers/gpu/drm/bochs/*.ko /lib/modules/4.18.0-12-generic/kernel/drivers/gpu/drm/bochs
depmod -a
modprobe bochs-drm
systemctl start sddm

And then I got SDDM back on screen !

I hope bochs-drm will be included in future standard kernels, as asked for in bug 1795857, because until it is included, these steps will be necessary after each update of the standard kernel.

Conclusion

That was not a simple solution to my problem. Did it work ? Yes ! Do I recommend it to anyone ? No, this method is too brittle and takes too much time.

But it shows how Open Source makes it possible to solve your problems without the help of the author or packager of the software you want to use, and even do things that where not anticipated by them.

Sorting files by date

Category : Bash Linux

My phone saves pictures with names in the format

IMG_YYYYMMDD_HHMMSS.jpg

I upload all images to my machine in a single directory, but I want to sort them by date. I use the following commands to sort them, in two phases :

First, create the directories

ls|cut -d_ -f2|sort -u|while read date; do
mkdir -p /data/pictures/${date:0:4}/${date:0:4}-${date:4:2}-${date:6:2};
done
  • ls gives all image names
  • cut -d_ -f2 extracts the date of each picture
  • sort -u removes duplicate dates with the -u (unique) option
  • while read date loops on all dates to run the next command
  • mkdir -p creates the directory for the date, and eventually the year
  • The syntax ${date:0:4} extract parts of the date. In this case the year

Then move images into directories

ls|cut -d_ -f2|sort -u|while read date; do
mv *$date* /data/pictures/${date:0:4}/${date:0:4}-${date:4:2}-${date:6:2};
done

Only the command inside the loop is different :

  • mv *$date* moves all pictures of the same day in the target directory

If the date is not in the file name, but is available in the directory, you can get it with the command ls --full-time and parse the output, or you can use the command stat to get exactly what you want, as it gives you full control on the output.

Converting a Python project to PEP8 with Shell

Category : Linux Python

When I started Python I didn’t learn PEP8 immediately so my code was not following the standard format. As I know Bash quite well, I’ve converted my projects later, with the following commands.

Convert tabs to 4 spaces

find . -name '*.py'|xargs pep8|grep W191|cut -d : -f1|xargs sed -i "s/\t/    /g"

Suppress spaces when they’re alone on the line

find . -name '*.py'|xargs pep8|grep W293|cut -d : -f1|sort -u|xargs sed -i "s/^ *$//"

Suppress spaces at end of line

find . -name '*.py'|xargs pep8|grep W291|cut -d : -f1|sort -u|xargs sed -i "s/ *$//"

Add spaces after “:”, “,”

find . -name '*.py'|xargs pep8|grep E231|cut -d : -f1|sort -u|xargs sed -i 's/:\([^ ]\)/: \1/g'
find . -name '*.py'|xargs pep8|grep E231|cut -d : -f1|sort -u|xargs sed -i 's/,\([^ ]\)/, \1/g'

Suppress spaces before or after {} [] ()

find . -name '*.py'|xargs pep8|grep E201|cut -d : -f1|sort -u|xargs sed -i 's/{ */{/g'
find . -name '*.py'|xargs pep8|grep E201|cut -d : -f1|sort -u|xargs sed -i 's/\[ */[/g'
find . -name '*.py'|xargs pep8|grep E201|cut -d : -f1|sort -u|xargs sed -i 's/( */(/g'
find . -name '*.py'|xargs pep8|grep E202|cut -d : -f1|sort -u|xargs sed -i 's/ *}/}/g'
find . -name '*.py'|xargs pep8|grep E202|cut -d : -f1|sort -u|xargs sed -i 's/ *]/]/g'
find . -name '*.py'|xargs pep8|grep E202|cut -d : -f1|sort -u|xargs sed -i 's/ *)/)/g'

Suppress spaces before “(“, “:”

find . -name '*.py'|xargs pep8|grep E211|cut -d : -f1|sort -u|xargs sed -i 's/ *(/(/g'
find . -name '*.py'|xargs pep8|grep E203|cut -d : -f1|sort -u|xargs sed -i 's/ *:/:/g'

Add a blank line at end of file

find . -name '*.py'|xargs pep8|grep W292|cut -d: -f1|while read f; do echo >> $f; done

Add a blank line before “def” or “class”, to be run two times

find . -name '*.py'|xargs pep8|grep E302|cut -d : -f1|sort -u|xargs sed -i 's/^def/\ndef/'
find . -name '*.py'|xargs pep8|grep E302|cut -d : -f1|sort -u|xargs sed -i 's/^class/\nclass/'

Sometimes I don’t really want to cut long lines, so I keep lines up to 120 characters

find . -name '*.py'|xargs pep8 --max-line-length=120

I think that 80 is to restrictive and based on terminals of the last century or on paper sizes that no one uses any more.

However, I’ve learned how to cut long lines with chained calls.

Before:

compte.dernier_solde = SoldeJour.objects.filter(compte_id=compte.id).order_by('-date_solde')[0].solde

After, with parenthesis to tell the Python parser that a single expression is on multiple lines:

compte.dernier_solde = (
SoldeJour
.objects
.filter(compte_id=compte.id)
.order_by('-date_solde')[0]
.solde
)

I’ve also configured Scite, my favorite editor, for spaces and tabs.

use.tabs=0
tabsize=4
indent.size=4

A new Django widget

Category : Python

I’m building websites with Bootstrap and Django. I’ve already created a form on one of them with a file input. The standard file input can’t really be styled with CSS apparently, so I’ve searched for a way to have a nice Bootstrap file input and found this. In short, it uses a button and a text input for display, and a transparent file input above them to use the browser access to local files. Some Javascript makes those two parts communicate.

First I’ve used this manually in a form directly in the template. Today I discovered that you can write your own Django widgets, so I tried to reorganize this file input.

Creating the widget

First I created a widgets directory in my app, with the following files : __init__.py (empty) and styledfileinput.py.

styledfileinput.py contains :

# StyledFileInput
# coding: utf-8

from django.forms.widgets import FileInput
from django.utils.html import format_html
from django.forms.utils import flatatt

class StyledFileInput(FileInput):

class Media:
css = { 'all': ('styledfileinput.css', ) }
js = ('styledfileinput.js', )

def render(self, name, value, attrs=None):
if value is None:
value = ''
final_attrs = self.build_attrs(attrs, type=self.input_type, name=name)
if value != '':
# Only add the 'value' attribute if a value is non-empty.
final_attrs['value'] = force_text(self._format_value(value))
return format_html(
'<div class="input-group">\n'
+ '<span class="input-group-btn">\n'
+ '<span class="btn btn-default btn-file">\n'
+ 'Browse... \n'
+ '<input multiple="" {} >\n'
+ '</span>\n'
+ '</span>\n'
+ '<input class="form-control" readonly="" type="text">\n'
+ '</div>\n', flatatt(final_attrs)
)

I’ve created a StyledFileInput class inheriting from FileInput, and redefined the render function with the whole HTML block. I’ve kept the attributes and value handling, so that Django can use it properly.

There is an inner Media class to define the required static files. This two files are placed in the static directory of my app.

Here is the content of styledfileinput.css :

/* file buttons should be styled */
.btn-file {
position: relative;
overflow: hidden;
}

.btn-file input[type=file] {
position: absolute;
top: 0;
right: 0;
min-width: 100%;
min-height: 100%;
font-size: 999px;
text-align: right;
filter: alpha(opacity=0);
opacity: 0;
background: red;
cursor: inherit;
display: block;
}

Then styledfileinput.js :

(function($){
$(document)
.on('change', '.btn-file :file', function() {
var input = $(this),
numFiles = input.get(0).files ? input.get(0).files.length : 1,
label = input.val().replace(/\\/g, '/').replace(/.*\//, '');
input.trigger('fileselect', [numFiles, label]);
});

$(document).ready( function() {
$('.btn-file :file').on('fileselect', function(event, numFiles, label) {

var input = $(this).parents('.input-group').find(':text'),
log = numFiles > 1 ? numFiles + ' files selected' : label;

if( input.length ) {
input.val(log);
} else {
if( log ) alert(log);
}

});
});
})(django.jQuery);

You can see that this code is wrapped in a function called with django.jQuery for $. This is necessary as I also use the AdminDateWidget, which uses the following JavaScript :

/* Puts the included jQuery into our own namespace using noConflict and passing
* it 'true'. This ensures that the included jQuery doesn't pollute the global
* namespace (i.e. this preserves pre-existing values for both window.$ and
* window.jQuery).
*/
var django = django || {};
django.jQuery = jQuery.noConflict(true);

Using the widget

Now I can use it in a Django form :

from app.widgets.styledfileinput import StyledFileInput

class DocumentForm(forms.ModelForm):
class Meta:
model = Document
fields = "__all__"
widgets = {
'date': admin.widgets.AdminDateWidget(),
'filename' : StyledFileInput(),
}

In the template, nothing changes in the form itself :

<form method="post" enctype="multipart/form-data">
{% csrf_token %}
<div class="panel panel-default">
<div class="panel-body">
{{ doc_form|bootstrap_horizontal }}
<button class="btn btn-default" type="submit">Envoyer</button>
</div>
</div>
</form>

You just have to be sure to put somewhere in your template {{ doc_form.media }} so that the required static files are included.

Conclusion

Now that the Bootstrap file input is redefined as a widget, it’s easy to attach to any classic Django forms, and it’s easy to reuse in any project.

If you have any suggestion to improve it, feel free to comment.

Open source software I use

Category : Non classé

I’ve chosen some open source software to use daily. Besides being open source, I’ve some more criteria :

  • They should run on Windows and Linux, and sometimes Android
  • Or it should be a web application, with a nice interface on mobile
  • Or it is used to bridge the gap between different systems

Recently I’ve included a new criteria, about data and configuration accessibility : they should be shared between devices.

Multi-systems applications

  • VLC
  • Firefox
  • Libre Office
  • Gimp
  • FreeFileSync
  • Games : OpenTTD, MAME, DosBox, FreeCiv
  • Development : Python, Subversion, Scite, MySQL, Apache, KDiff3
  • VirtualBox

Web applications

  • TT-RSS, with mobile application
  • Pydio, with mobile application
  • Trac
  • WordPress 🙂
  • PHPMyAdmin

Bridges between systems

  • Samba
  • Putty, Mintty
  • Cygwin
  • WinSCP

Cloud services

These applications are so powerful, I don’t know how I could set up a system to replace them, let alone maintain it in the long run.

  • GMail
  • Google Agenda

Shared data and/or configuration

  • Firefox
  • TT-RSS
  • Pydio
  • GMail
  • Google Agenda

 

Search

Recent Posts

Categories

Meta

© Copyright 2020. All Rights Reserved