Automatic bug report and stack traces for GIMP

While I was working on yet-another-crash without a backtrace, I realized that we could just generate automatic backtraces upon crashes and tell people about it. This is how I ended up writing a debug tool for GIMP, popping-up a dialog with a nice text encouraging to report bugs. You’ll notice that the main text is non-technical. The goal is not to display non-understandable error messages which nobody will understand. All the technical part is in the below section and is just to be copied by a single button click and reported to us verbatim. 🙂

This technical part contains: GIMP version (and commit information if available), compiler, main dependency version, and finally the errors and backtraces of these errors.

Note: this doesn’t “report” the bug on your behalf. Anyone still has to make the conscious action and go on our bug tracker. But we make things easier and just a few buttons and a copy-paste away.

Someone asked me if I could make a blog post about it, so here it is.

How does this work?

Used to be based on glib…

We already had some backtracing capability in GIMP, mostly using glib API g_on_error_stack_trace(). The main problems of this API:

  • that this function outputs to stdout (which means that you needed to run GIMP from terminal to get the trace, and until now this was only used with specific command line options or on unstable builds);
  • sometimes it was not working for weird reasons;
  • it works only in Unix-like operating systems (in particular not in Windows);
  • it is based on gdb only (as I soon discovered)

So I ended up looking what this function was doing. As I said, the basics is that it simply uses gdb if it is installed on the machine. I am still unsure why, but it was doing so using the interactive mode, therefore entering commands through the standard input with a pipe. Why is it weird? Because gdb has a batch mode especially done for such non-interactive calls. I suspect actually that some of the times g_on_error_stack_trace() failed to work correctly was maybe because it was stuck (but I am not sure, I have not tried to dig much more, so maybe I say shit). But the worse issue was that it was simply printing to stdout. So if I wanted to get the output inside a string in order to use it in the graphical interface (we should not expect people to run GIMP from a terminal!), I had to do more piping of the output. Well at some point, that was just ridiculous to stack processes one after another after another after…

… then based directly on GDB…

This is how I started to reimplement the feature. I simply run gdb in batch mode, and I keep the result in a string for later display in a dialog. This was actually very straightforward. See commit bb88a2d52f.

This also allowed me to get a slightly better stacktrace since I could customize the command. So I request “backtrace full“, getting us local variable contents.

… and LLDB…

Then I remembered that some bug reporter on macOS was using lldb, the debugger from the LLVM project. Since LLVM is default on macOS, I assumed that LLDB is much more common there than gdb too. So I added support for it. This was quite easy too, I just had to search for command equivalency. See commit 4ca31b0571.

… and finally the GNU libc!

Finally I was told of the backtrace() and backtrace_symbols() API. This seems to be a GNU-only API (man says these are GNU extensions). Anyway this should make these always present on common Linux distributions, which is very good news. It means that we will always get “something” on Linux (also the result is much quicker than calling gdb or lldb). Unfortunately the output of  backtrace() is not that exhaustive: basically you get function names, and in particular neither file name nor line number even if you built with debug info, nor variable and parameters contents. So it’s a bit less useful. Yet it’s better than nothing! See commit 4fd1c6c97c.

So in the end, my tool tries gdb, then if absent, lldb, and finally fallbacks to backtrace() if available. This should hopefully gives us traces of crashes and errors in most cases!

The difficulties

Issue 1: do not rely on memory allocation after a crash

There were still a few issues. One of them is that you may notice that I use this dialog for 2 kind of errors: fatal errors (crashes) and non-fatal errors (WARNING, CRITICAL, etc.). l use the same code, but while testing, I realized that I often could not create the dialog from the main process when GIMP crashed. In Linux at least, once the program crashed, I was able to catch the terminating signal enough to do last minute actions, but it seems allocating more memory was not amongst the possible actions (that was my assumption based on tests, I may be wrong, don’t take this for manual talk). Well I guess that makes sense to forbid more memory management, especially if the crash is related to memory bugs. This means that even just creating a new dialog is not possible (requiring allocation of a new GTK+ widget).

This is why when crashing, I run the dialog as a separate process, whereas I run it from within the main process for non-fatal bugs.

Issue 2: backtrace() needs to be run by the main process

When running as a separate process, should the back trace be generated by this other process or from the main process? At first it made sense to have it generated through the new process, but then this has 2 inconveniences:

  1. I am duplicating the back-trace generation code (since I sometimes need to run it from within GIMP, sometimes from outside) and code duplication is never good (even maintenance-wise, you end up with different version. This sucks). You can make common core code as exception, but it’s just not ideal (it makes the build rules complicated).
  2. From the outside process, I can attach to the main process with gdb or lldb but I cannot use backtrace()anymore. That would mean that a lot of people would not get the auto-generated traces (not everyone installs a debugger!).

This is why I decided that the backtrace is always generated by the main process and in case of a crash, it is passed along through a file, instead of a parameter. I could have piped it which would have been just as easy, but Dr. Mingw (see below the Win32 section) was already using a file. So I chose to do the same to be as consistent across platforms as possible (also a file has some advantages: in the extreme case where the dialog breaks too, we may ask a bug reporter to look if a file has still been generated with the info).

Also since — as I said in issue 1 — memory allocations are more likely to fail during crash handling, you need to use backtrace_symbols_fd() instead of backtrace_symbols().
The _fd() variant is guaranteed to run without memory allocation (this is written in the man). And now we have traces on most systems, still with GNUlibc fallback!

Issue 3: error avalanches

Another issue is that, in case of non-fatal errors, you may often have a few of them one after another. Sometimes they may be generated as dominos (you get the second as a consequence of the first error), sometimes it’s because of long-running operations which would just reproduce the same errors many times.

Worse case scenario: a long-time contributor, Massimo, directed me to a bug which would output dozens of thousands of errors in a few seconds. Actually that depends on the size of a selection, and in some of my tests, I had hundreds of thousands of errors!
Obviously you don’t want to create a dialog each time (this example was not even a bug which crashes GIMP, but creating hundreds of thousands of dialogs may do the killing job!). So you have to just update the current dialog with additional errors. But even doing so is very time consuming. Updating a dialog hundreds of thousands of times in a few seconds is at least likely to freeze the whole GUI for a dozen of minutes (I know, I tried!).

So I decided to limit the backtracing, but even the error handling. In a single dialog, I add up to 3 backtraces and 10 errors at most. Any more errors would just be redirected to stderr.

Issue 4: debugging preferences

Moreover do we want the dialog to appear for every kind of errors? In particular, we have WARNING, CRITICAL then all fatal errors. CRITICAL are usually really bad, so we definitely want debug info here. But what about WARNING? I mean, they  are bad too, and they are signs of a bug somewhere. But these are more minor bugs, sometimes also bugs on external data which we warn about (and have no control on). Also we often output warnings when we encounter bugs in other software (for instance, one of the recent bugs where my dialog worked was on a bug in KDE’s API for color picking, and there is not much we can do about it in GIMP but report upstream). So I added finer-grained settings, because you certainly don’t want to make creating with GIMP painful if it pop-ups errors every few hours!

Actually it is even possible to disable all debugging through GIMP preferences, even during crashes, if someone is really not interested at all in reporting bugs, hence contributing to GIMP improvement.

Note: on Windows, the debugging preferences page doesn’t exist at all because the backend we use is not customizable anyway. See dedicated section below.

Issue 5: multi-threading

As explained, we don’t only handle crashes, but also runtime errors. Since GEGL is so close to the GIMP project, it made sense to handle its errors as well (actually long-term, it would make sense to handle errors from any dependency, but let’s do it step by step). So I also catch GEGL’s WARNINGs and CRITICALs. But then I realized that since GEGL uses a lot of multi-threading, getting a backtrace from the main thread when the error happened in another was completely useless.

This combined to the fact GTK+ code must be run in the main thread, therefore to create or update my debugging dialog, I need to pass the information from the thread where the bug occurred to the main GTK+ thread. This can be done with gdk_threads_add_idle_full(). This call obviously adds a delay so you’d end up getting traces from the wrong code, and after an unknown delay. This is double useless.

As a consequence, to handle multi-threaded debugging, I needed to make sure that the stack trace was generated from the thread the error happened, without any delay, and only then it could be sent to the main thread with an idle function.

Issue 6: the tweaking

Then you have all these little details to make the experience not too terrible (at least I am not saying we should make it a good experience, a bug is never a good experience! ;P).

For instance handling a crash, I add a button “Restart”, allowing — as the name implies — to at least restart GIMP immediately.

When non-fatal bugs are reported, we should advise people to save their images and restart GIMP (of course, for crashes, they won’t have the possibility to save themselves, so don’t make them sadder by reminding them).

Also I have to be extra careful to not generate new WARNING or CRITICAL from within this code because then you could create cyclic calls. You don’t want to end up crashing the software because of the debugger which initially fired up only for a minor bug.

Well you get the idea! These are the kind of tweaking you just discover as you implement such a system and you have to take care of them as you go on.

Future work

Something we have been discussing would be to save the opened images in backup files upon crashing. Of course with some kind of crashes, it may not be possible, but that is worth trying at least!

I’ve actually started working on it (with commit d916fedf92  from yesterday). As expected, it’s working most of the time, but while testing various crash conditions, I had some cases where last-second backup failed. I have not dived into the code yet to understand why and what, and if there is a solution to these.

GIMP is quite stable now (at least on GNU/Linux), and quite rarely crashes (well I say this but we had some instability these last few days because of core changes in selection and channels so the auto-debug dialog was very useful). But for this one time when it happens, handling it the most gracefully possible implies saving the current state of work. Then obviously next step will be to propose recovery on next GIMP start.

More on this later as I will continue working on it…

What about Windows?

Now the last remaining issue is Win32! Having GDB or LLDB there might be possible (I have not checked) but probably not the best path. It turns out a contributor, Mukund Sivaraman, did already add support for backtrace generation on Windows upon a crash, back in 2015. This is using the ExcHndl library from the Dr. Mingw project. Basically this is extremely easy to use since there are only 2 functions in the API: one to init the library, one to choose a file where the backtrace will be outputted.

void ExcHndlInit(void);
bool ExcHndlSetLogFileNameA(const char *szLogFileName);

So yes, since 2015, backtraces were simply outputted into a file somewhere, and people just never knew where and how to find it. What I did was simply to piggy-back on this feature, grab the backtrace from the generated file, and display it in our GUI. And that’s it!

Since I needed my own code to run after Dr. Mingw, I had a look how this tool actually made its job. In its code, I saw it was using  SetUnhandledExceptionFilter()to run its action just before the crash. What I did was adding another exception handler with the same function, but registering my handler first beforeinit() Dr. Mingw. This way Dr. Mingw call my handler immediately after its own because it keeps track of any handler previously set and call it after itself.
See commits ae3cd00fbd and 4e5a5dbb87.

Now this has a few limitations: the backtrace generated by Dr. Mingw is not that complete compared to a good gdb backtrace. Also sometimes, I had some crashes which this tool would not catch. I am no Win32 expert and did not spend much time on it, so I don’t know why.
Finally this works only on crashes, in particular I cannot generate backtraces on a whim as I can do on other platforms, which allows to generate backtraces even on WARNINGs or CRITICALs messages for easier debugging, even without a crash.

Well in the end, Win32 always ends up less featured and most annoying to debug. I guess there is nothing to be done since I remind we are still looking for Win32 developers on GIMP. We have had very few contributions of Windows developers for all the years I’ve been around, quite sadly! If you are interested to contribute on this cool piece of software, be very welcome!

We got our first reports with automatic traces!

Even though the tool is still only present in the development version, some people build GIMP from master, and we already got a few bug reports with traces included directly! This is very cool.
Actually even Aryeom got such dialogs, which resulted in some bug fixes already (and more to come)! 🙂

So yeah when I fixed my first bugs thanks to these automatically generated back traces, that made me happy because I felt this new tool will make life a lot simpler and I knew my time was well spent. 😉

You’d think a developer of GIMP would not be happy to get a back trace. And yeah, I’d prefer that GIMP was perfectly bug-free. But there is no such things, and as long as we get bugs, we may as well get well-illustrated reports to easily fix them. This is why I am happy! We are constantly on our way to a much more stable GIMP.

Yeah!

Reminder: my Free Software coding can be funded on:
Liberapay, Patreon or Tipeee through ZeMarmot project.

Ibus-Hangul and Compose key: the incredible journey of a simple patch

Today I decided to tell how I reported a bug (then ended up fixing it) on a non-GIMP related project. Well I do regularly this kind of stuff, and this could have just been one more of these silent commits to a random project as I did many times in my life. But since I decided recently to post more articles, well… I may as well tell a story as one-time contributor (as opposed to “regular contributor”) for once!

Also I think the whole process of reporting a bug on projects you don’t know at all — worse! A whole stack of software you don’t know much! — is quite interesting for people wondering how they should report bugs happening to them.

Finally another reason is to advertise a bit my work, since I remind I am trying to get my Free Software coding crowdfunded through ZeMarmot project.  I am hopeful these kinds of side-stories highlight how our project is useful for Free Software as a whole, not just GIMP. Because yes, that seems unrelated, but in the same time, if I can be funded to hack Free Software full-time, it means more such unrelated bugs can be fixed! So you can consider that this kind of patch is also funded when you crowdfund ZeMarmot. 😉

The context

My main input is the ibus-hangul (Korean) input engine,  and the main languages I write are English (ibus-hangul is basically the same as “English (US)” layout when in LATIN mode, which is my default mode) and French! Sometimes when needed, I can easily switch to writing Korean while keeping the same layout.

To write French, I simply mapped a Compose key on the CapsLock key (which I don’t need). Why not switching to a layout with French characters? Well mostly because I don’t like the default French layout and am used to the US one (I have used it for a few years now), and I am also used to using Compose (I can write quickly enough all French characters with Compose, most people would not know the difference in term of typing speed).
Moreover I don’t mind changing layouts but this can be a bit confusing at times, especially if you need to change your layouts every few minutes. It’s better to just keep the same if possible.

Finally this is also the input which Aryeom uses (her main language being Korean) and she writes both in Korean and in French on daily use.

The route from bug to patch…

Or: how to report a bug … when you know very little about a software stack of dozens of software (really device input goes through so many layers that it’s making me crazy!)

Step 0: the bug!

We were using successfully our setup up until Fedora 25: Korean’s ibus-hangul IME with a Compose key. This Compose key settings was done through GNOME Tweaks Tool.

 

Then some fateful day, we updated to Fedora 26. And paf! The Compose key stopped working. Oh fateful day!

We hoped that it would be fixed by Fedora 27 (since we updated late to Fedora 26, Fedora 27 release was just a month away), but it didn’t.

Step 1: I click this button and it doesn’t work => GUI problem?

Back then though, I didn’t link the bug to ibus or ibus-hangul yet.

I first tried to track this issue in GNOME Tweaks tool (hoping that might have just been just a GUI bug; by the way, now the bug is fixed and the issue was not in Tweaks, but it seems I cannot close the report in gitlab and nobody is answering! If someone from the project reads me: just close the report, please) since that’s where I was setting the feature.

Of course, my secret hope was to just see other developers take care of it and not having to dive into unknown code. Don’t get me wrong, I’d love to fix every bug I encounter myself, but I also have a limited life span! ;P
Also I am lazy. I hear that’s a good trait by the way, nothing to be ashamed of. So really hoping someone else does the job at your place is my best expectation.

Step 2: going deeper into layers

But then after waiting for more than a month with a broken compose key, I couldn’t stand anymore to write my French texts without accents. So I investigated a bit more by myself and whistled in the wind (as you can see, I still haven’t had a single answer in 3 months on this first bug report. Bit sad 😞), looking for which dconf key Tweaks Tool was changing, and seeing that it worked at least here: the dconf key was properly updated so Tweaks was likely not at fault.

So was GNOME the culprit not doing proper action with this dconf value? To decide this, after I understood that this was just a X11 options (the bug also happened on Wayland though), I tried various combinations of trying to set “compose:caps” directly with setxkbmap, or even updating XKBOPTIONS directly inside X11 config files (something I hoped I would never have to do ever again, and for years it seemed like it would hold) and restarting X.

It didn’t work. So I became more certain the problem may be deeper and something was broken in X and Wayland. I asked someone who knows better than I on this whole graphics desktop stack (Thanks Carlos for always helping me out! I know I can be annoying sometimes! 😛).  He advised me to open a bug report at libxkbcommon, which was used both for X11 and Wayland. So I did.

Step 3: …and deeper…

Continuing my own investigations, I realized that the problem was not happening on every input method, but even stranger, the input order changed the behavior! 😥

This is when I was told the order indeed matters and that’s expected, which I find a bit weird since it makes for very particularly unique bugs. But anyway I was not here to argue about these kind of design decision. So I followed libxkbcommon‘s developer’s recommendation and opened another report at xkeyboard-config instead.

Step 4: and back up!

Then I left time pass again for about a month, but once again failed to get an answer, therefore tried to investigate more myself, once again. I reread and thought back about my previous investigations and realized that the broken inputs were all Ibus ones (it turns out I only use Ibus inputs nowadays, mainly the Korean or sometimes the Japanese one). For people wondering what is Ibus, this is the input framework for writing languages complicated enough for which just a layout change would not work (for instance Japanese, Korean, Chinese, etc.). This is when I first started to be on the right path.

But is the issue in Ibus or specifically Ibus-Hangul?
To answer this question, I thought to be clever (I wasn’t necessarily!) by checking Fedora repository contents. There I realized that Ibus-Hangul package didn’t change its version (1.5.0) from Fedora 25 to 27. Since my issue appeared in Fedora 26, I went with the assumption that the broken part was most likely Ibus (which indeed upgraded from Ibus 1.5.14 to 1.5.16). So I opened a bug report for ibus.

This time, I had a very quick response (thanks Takao!), then a few exchanges not only telling me that the issue was indeed triggered by a change in Ibus, yet that was not really a bug in Ibus (well it’s a bit sad that Ibus does not guarantee compatibility, but I can also understand why). Ibus-Hangul had to be updated to follow Ibus changes. Better, the developer of Ibus gave me the necessary hints to update Ibus-Hangul.

Step 5: the patch

And then a few hours later, I was able to clone, read, understand and finally simply patch myself Ibus-Hangul. I immediately installed it for myself and for Aryeom, then I went and did a pull-request on upstream project (⚠ important: never forget to contribute your patches upstream! I know, it means a bit more work; sometimes making clean code which will be accepted upstream may even take longer than making the quick-and-dirty fix for just yourself. Yet that’s how the whole software ecosystem will improve! Imagine how bugged Free Software would be if others were not contributing their patches!).

Hopefully my pull request will be accepted soon even though ibus-hangul activity seems a bit slow (yet existing! There were commits 2 months ago). But anyone in our case reading this, you can already build and install with my patch if you need.

This is also a good example to work from if ever you use another Ibus input method with the same problem. That should be quite easy to update all Ibus engines hopefully.

Step 6: it’s never finished!

The issue is mostly fixed, at least on Ibus side apparently. There is still some remnant of issue, which is that my remapped CapsLock was now working both as a CapsLock and a Compose key which is obviously not the expected result! So if I want to write ‘œ’, it would output ‘Œ’ and locks capitalization. That’s not good…

Getting again a bit of help from Takao Fujiware from Ibus project (really thank you a lot!), it seems that the bug is in GNOME this time, since I can work around it by running directly a command line command:

setxkbmap -layout us -option compose:caps

Also I realize that the issue happens even with non-Ibus inputs this time. So somehow it would seem that GNOME may not set properly the option? Let’s wait and see for answers on this report. At least this time, I have a workaround (I even have 2 different workarounds, but let’s not complicate things!)! 🙂

Conclusion

On our beloved GNU/Linux operating systems, the text input situation is mostly ok, though sometimes I must admit I wonder how many people use slightly more exotic input settings since I regularly encounter funny bugs. Well I have lived in Japan for 2 years and in Korea for 1 year, so I also know that Free Software communities are unfortunately less common over there. And among these communities, people also needing a Compose key must be even rarer.

As you may know, I love languages, so I really hope that things will improve regarding any language support. 🙂

As for the whole route from a bug to a patch, it is interesting to see how it took 5 bug reports to finally get to the right project, and a little more than 3 months for a patch written in a few minutes once I knew what this was about. Somehow this is part of the “unseen”, slow, lengthy and boring work done by every contributors out there (even more one-time contributors since they start without the whole package). And no matter how good you may think you are, you are always a newbie somewhere. I personally even consider to be a constant beginner on most topics, even the ones I know best. This makes things easier because I don’t set impossible expectations on myself but also others.

Thanks for reading my story!

Reminder: my Free Software coding can be funded on:
Liberapay, Patreon or Tipeee through ZeMarmot project.

New “mypaint-brushes” package

Since January 1st, GIMP depends on the mypaint-brushes” repository which I am maintaining until MyPaint project finally takes it alongside its other repositories.

I am hoping that I won’t have to maintain this for long and am looking forward for the MyPaint developers to take care of it (and last I heard of it, in the bug report, they wanted to). So this blog post is also to say that I am not trying to fork MyPaint or anything. 😛 I am just taking a little advance because we cannot wait much longer unfortunately since GIMP now uses libmypaint and we are really looking into releasing GIMP 2.10 as soon as we can. Therefore we need to have MyPaint brushes as their own separate package, which will allow:

  • Not having MyPaint as a de-facto dependency of GIMP since brushes are currently part of MyPaint itself (not separate) and our new MyPaint brush tool in GIMP is basically useless without the brushes.
  • We can now check the existence and path of the brushes at build time (through pkg-config), instead of guessing, making sure GIMP will have brushes to work with.
  • The libmypaint library has been recently versionned because of changes in its API (current libmypaint repository’s master branch is the future version 2). Since libmypaint 2 has no release yet, GIMP uses libmypaint 1 (and likely still will when we will release).
    Similarly the mypaint-brushes package I created is also versionned because the brush format has evolved with the API. It has new settings and new inputs, which were crashing older libmypaint. New settings crashes have been fixed, but the new input crashes still exist to this day. Obviously this is not good and we cannot afford GIMP to crash when using newer brushes. So we need versionned brushes.
  • Even if the crashes were all fixed, some brush settings changed their behavior/meaning (for instance the computation of the speed). That means basically that the brush format changed in a non-compatible way. Another reason to version brushes.

So that’s a new package out there, but an important one and any distribution out there which wishes to package GIMP will also have to package it.

Also if other projects use libmypaint, I would suggest you also depend on mypaint-brushes (as should MyPaint itself actually). 🙂

P.S.: of course, this does not fix the custom brushes that someone would import from MyPaint to GIMP, but at least we’ll have good default brushes.

P.P.S.: if you build yourself the development version of GIMP, check out the INSTALL file carefully. In particular, you don’t want to install the master branch (as I said above, master is version 2, we use version 1) and when installing in a non-standard prefix, make sure your PKG_CONFIG_PATH environment variable is properly set.

Reminder: my Free Software coding can be funded on:
Liberapay, Patreon or Tipeee through ZeMarmot project.

ZeMarmot: GIMP 2.9.8 and end-of-2017 report

Here it is, GIMP 2.9.8 has been released some days ago now, the latest development version of GIMP! As it is customary now, let’s list our involvement in this version so that our supporters on crowdfunding platforms know what they funded. 🙂

Since it also happens to be the end of the year, I complete this post with our end-of-year report, as we also did in 2016.

What we did for GIMP 2.9.8!

During this release span, I focused most of my efforts on bug fixing. I finished a few features here and there but actually even restrained myself from coding too many new stuff! Why? Because I believe we have enough and at some point, we should just release GIMP 2.10. Of course, GIMP 2.10 could be even twice as awesome if we push it by a few more months, and 5 times more awesome with even more time. But then in the end, if you never see it, what’s the point, right? Actually I even plan on just doing this (bug fix and finishing what was started) until we get 2.10 out. Let’s stop feature craziness!

Apart from a lot of bug fixes, I did a lot of bug triaging these last months (looks like I participated in 122 bug reports between 2.9.6 and 2.9.8, i.e. 3 and a half months).  And this month, I also reorganized our bug tracker still for the same reason (pushing GIMP 2.10 release forward) by reviewing the 50+ bugs we had in the GIMP 2.10 milestone to set as blockers only the ones we should really look into. Right now that’s down to 25 such bugs!

I also put some efforts in our stable flatpak release, which is how since October 16, GIMP has officially had its flatpak package on flathub! It is of course visible on the GNU/Linux section of GIMP’s download page with a nice “Install GIMP flatpak” orange button (notice also the cool drawing on this page? That’s Aryeom’s!).

Right now, you can only install the stable release, i.e. GIMP 2.8.x (flathub only accepts stable builds) but if you get it there, when GIMP 2.10 will be out, you will automatically get an official update!
In any case, this flatpak thing (in particular keeping our development flatpak manifest up-to-date with git code and testing the builds) is taking a lot of maintenance time!

As a whole that was 122 commits authored by me in GIMP repository between GIMP 2.9.6 and 2.9.8, out of 474 commits (so ~25%), and I pushed a few more commit from third-party when I reviewed them…
We also had again 2 guest commits by Lionel N., board member of LILA association, the non-profit managing ZeMarmot project.

Of course, though I said I focused on bug fixes, there are still a bunch of cool features I participed to during this release:

  • Support of password-protected PDF for import (the 2-commit feature implemented by Lionel from LILA!) and new procedure `file-pdf-load2()` API for plug-ins and scripts to open password-protected PDF files, but also multi-page PDFs (loading a multi-page PDF was already possible through the GUI but not by scripts and plug-ins).
  • Help system improvements: upon detection of locally installed manuals in several languages, GIMP will now allow selection of the preferred manual language in the Preferences dialog (Interface > Help System). I felt this was an important feature because we regularly had people not understanding why the manual they installed was not seen by GIMP. And they were right, especially since we don’t have as many manual languages as GUI languages. For example, we have 3 Chinese translations (zh_CN|TW|HK) but only a zh_CN manual. I could definitely imagine someone with a zh_HK GUI to go for the zh_CN manual as a fallback.
  • Verbose version (command line: gimp -v) now displays C compiler information (useful for debugging).
  • Canvas rotation and flipping information are now visible in the status bar, and this information is interactive (clicking the flip icons will unflip the canvas; clicking the rotation angle opens the “Select Rotation Angle” dialog). Some people were indeed noting that with the ability to flip/rotate the canvas, in some cases, you may end up “lost” on whether it is currently rotated, flipped or whatnot. After all, the status bar already has zoom information and flip/rotation is quite a similar feature. 🙂
  • Screenshot implementation for KDE/Wayland.
  • Color picker implementation for KDE/Wayland.
  • Improve delay handling for screenshots.
  • Review HGT support patches and improve a bunch of stuff with auto-detection of the format variants (SRTM-1 and SRTM-3), and also a `file-hgt-load()` API for scripts and plug-ins.

But really, as I said, I think my bug fixes and maintenance of previous code was actually much more important than this above list, even though it is so less fancy (and I am a bit sad I cannot list bug fixes in a non-boring way!). And I will just focus more and more on fixes and stability to get GIMP 2.10 out as soon as possible.

ZeMarmot in 2017: our report!

You know it, ZeMarmot is not only about GIMP, even though this software is a huge part of it! ZeMarmot is about making an animation film in 2D drawing, traditional animation yet with digital means (i.e. drawing on computer, not paper). We draw with GIMP. Well Aryeom Han, animator and animation film director does so (not me). And we crowdfund this project.

Financial status

This year was a bit tough mentally and we really started to wonder if this project was a good idea for our lives. Project finances increased continuously yet very slowly, and were still extra low all throughout 2017 (under 400 € a month).

In October, I finally shouted a cry for help after my computer broke, and we are so happy that many people heard it! The funding increased by about twice.

Now let’s be clear! Our current funding is more or less 1000€ since October. This is a lot better than what we had before and it gave us a lot of hope. Yet it still does not pay full time salaries for 2 people (faaar from it, actually it cannot even pay a single full time salary obviously). So we still hope you will not forget us and if you appreciate our project and what we do, both on GIMP development and/or on ZeMarmot movie, please we will be very thankful if you can donate to the project.

ZeMarmot project donations can happen on:

» Liberapay «
(weekly funding, USD and EUR possible, lowest fees)

» Patreon «
(monthly funding, USD ($) only)

» Tipeee «
(monthly funding, EUR (€) only)

Live Streaming while GIMPing

We were conscious that the lack of news on the animation side was not the best. On the other hand, animation just takes time. That’s the way it is.

Depending on the complexity and details of the animation (as chosen by the team), a minute of animation can take a month of work or more (just search the web, all links say the same).

Of course, it depends on your artistic choices. If you do vector animation or Limited Animation (Simpsons or the likes), you can animate a lot faster. Basically you don’t take the same time to animate South Park or a Disney movie (which is not a problem, it’s a choice; I appreciate The Simpsons or South Park too). For ZeMarmot, as you know, we chose a detailed style with full traditional animation. At times we regretted this choice a bit but that’s the way things are.

That’s how Aryeom decided to live stream herself working! She took a few days to search software and found the Free Software OBS, understand how things work (well she also managed to break Fedora once by reinstalling NVidia drivers while following tutorials! :p), made many tests throughout December and since December 25, public livestreams started.

The work is regularly live-streamed at this address:

» https://www.youtube.com/c/LibreArtInfo/live «

Unfortunately we have not found the right organization yet to plan and give a schedule of future livestreams. So for the time being, the best is either to follow us on Twitter, subscribe to the Youtube channel,  or just try to have a regular look at above link.

Previous livestreams are recorded and listed automatically on the channel once the streaming ends, so you can also have a look later on older (not live anymore) streaming. Be aware though: this is real live of someone working really. That means it is real time, not accelerated 20 times (as all these speed painting you can see everywhere), errors may happen and are not edited out afterwards. There is no sound and Aryeom doesn’t interact with people. She is focused and doesn’t look at what is streamed (we will sometimes look at the chat though and may answer questions but don’t consider it as a granted “feature”; this is a peek at an animator work, not a service). The artist sometimes goes for a rest, and so on. It can even be boring at times. Also the longer recorded streaming in the list is more than 7 hours straight! Fair warning. 😉

Still I think that’s an awesome experiment and we already had some very cool comments, like people thanking us (some in English, many in French) because that it is a bit like being allowed to sneak into an animation studio to observe the animators working.

 

Art+Code symbiosis

We regularly have the question: “why don’t you have separate crowdfundings for development and animation sides?

Answer: because this project is a whole. It is symbiotic: I do Free Software because I use it; if I didn’t have ZeMarmot (or another project where we use GIMP), then I would likely not contribute to GIMP. It is that simple. Aryeom as well would likely not use GIMP if she didn’t have a developer by her side.

We remind that is how I started my first patches: because we had crashes and many issues with GIMP and this was not enough for us as a professional tool at the time. We are very happy to tell you that now, it is. Not only because of us, far from it, let’s be clear! We are so happy to hack together on GIMP with several very talented developers (among them, Mitch, GIMP maintainer who is still here after 20 years!). But we were allowed to do our part and this is the reason we stuck around.

To illustrate, just a few hours ago, thanks to Aryeom’s streaming of her work, we were able to have an unexpected live demo of how well we work together. During her live, GIMP crashed! Ouch! In a few minutes, she was able to find reproduction steps during the live streaming. Less than two hours later, I fixed the crash then improved my fix in the master repository of GIMP (it actually took even just a few minutes to reproduce and fix the crash, but well I also had other priorities which I could not drop immediately!).

That’s how well we work together and what you pay for when you donate to ZeMarmot project, and finally that’s the reason why it is a 2-people project, not 2 separate projects. 🙂

So if you ever hold your donation because you only want to pay for a movie, or at the opposite only want to pay for GIMP development, I hope you will review your judgment and see why you get to your goal even better by paying for both!

GIMP Motion

GIMP Motion is our plug-in for animation in GIMP (we talked about it earlier for simple then complex animations). You can also see it in action in Aryeom’s live streamings by the way, nearly daily now.

Unfortunately I have kind of neglected it lately, and that’s mostly for the reason I told earlier: because I am really focusing on getting core GIMP 2.10 out. So I hope this won’t just drag forever (seriously I want to get done with GIMP 2 and go forward with GIMP 3!)
That means that GIMP Motion will likely not be a part of GIMP 2.10. Yet it will be a part of a further GIMP 2.10.x release since we decided earlier that we would relax the no-feature policy on minor releases, which is how I decided that GIMP Motion was not ready to be part of a stable release. That’s exactly why I pushed for this no-feature policy relax for years (ever since 2014, cf. the section “GIMP Meeting(s)” on our LGM 2014 report!): so that we don’t have to rush half-done features nor push important releases forever.

Well we still use it internally, but that’s still very very rough and has many bugs. Be warned if you try it!

Documentating the process of animation

This year, we have been a bit light on documenting the process. Well we had a post on animatics, key-framing, etc. and one on background design. We also had a few talks during the NUMOK festival in Paris, as well as in the JM2L meeting in South of France where we could give some interesting details on the process as well which are not written here yet (but should be soon).

I am the first to admit that’s not enough since Aryeom and I really want to document the techniques behind giving life to still images. But as I said, we had been a bit down, overworked and penniless this year so this fell a little behind. Hopefully we’ll do better next year.

ZeMarmot in the news

This was also cool since, we got on local TVs twice this year thanks to JM2L! In all the videos, GIMP is clearly mentioned and shown on screen as well as Aryeom hacking animations on GIMP in GNOME. 🙂

We have been featured on France 3 (also in writing):

Then on PleinSud TV (at 2:32):

And a mention in a newspaper, Nice Matin:

New material

Thanks to the increased funding at end of year, we were able to renew a bit our material. In particular we bought a Wacom MobileStudio Pro (basically a laptop-tablet from Wacom) on which the first thing we did was to erase Windows and install a GNU/Linux (Fedora 27) and GIMP. And that worked well. We still opened more than a dozen of bug reports here and there, so don’t expect things to be perfect yet. But we are working on it!

We documented a bit our process on a Twitter moment and unfortunately had to take a pause because of hardware issues. We indeed had to send the tablet back to after-service, which gave it back after more than 3 weeks (2 days ago)!

Now we will have to start it all again. Be prepared, we might publish a very complete guide soon on how to get a very cool Wacom laptop with Linux and GIMP. 🙂

Conclusion

This year was hard but eventful, and the end of year gave us more hope after funding increased and we were able to upgrade our material.

The streaming of Aryeom GIMPing was also a very cool idea and we are happy to see that people seem to like it. To this day we only got positive feedbacks. We should have started this sooner!

We do hope that things will continue to improve. We love what we do and our project, and we really wish we will soon be able to say proudly that we are able to make a living by hacking Free Software and Libre Art.

When this day will come, this will just be a very very happy day. 🙂
Happy New Year 2018 everyone!

New format in GIMP: HGT

Lately a recurrent contributor to the GIMP project (Massimo Valentini) contributed a patch to support HGT files. From this initial commit, since I found this data quite cool, I improved the support a bit (auto-detection of the variants and special-casing in particular, as well as making an API for scripts).

So what is HGT? That’s topography data basically just containing elevation in meters of various landscape (HGT stands for “height“), gathered by the Shuttle Radar Topography Mission (SRTM) run by various space agencies (NASA, National Geospatial-Intelligence Agency, German and Italian space agencies…). To know more, you can read here and there.
HGT download source: https://dds.cr.usgs.gov/srtm/version2_1/
(go inside SRTM1/ and SRTM3/ directories for respectively 1 arc-second and 3 arc-seconds sampled data)
You probably won’t find other links of interest since not everyone can do such data (not everyone has satellites!).

Here is what it can look like after processing: left is an image obtained from NASA PDF, and right is the same data imported in GIMP followed by a gradient mapping.

So the support is not perfect yet because to get a nice looking result, you need to do it in several steps and that involves likely a bunch of tweaking. My output above is not that good (colors look a bit radioactive compared to the NASA one!) but that’s mostly because I didn’t take the time to tweak more.

And so that’s why I am writing this blog post. Someone trying to import HGT files in GIMP may be a bit disconcerted at first (so I’m hoping you’ll find this blog post to go further). At first you’d likely get a nearly uniform-looking grey image and may think that HGT import is broken. It is not.

What’s happening? Why is the imported HGT all uniform grey?

GIMP by default will convert the HGT data into greyscale. That is not a problem by itself since we can have very well contrasted greys. But that doesn’t happen for HGT import. Why?

HGT contains elevation data as signed 16-bit integers representing meters. In other words, it represents elevation from -32767 m to 32767 m (with an exception for -32768 which means “void”, i.e. invalid data; since that’s raw data with minimum processing, it can indeed contain errors). Therefore once mapped to [0; 1] range, color 0 (pure black) is invalid, ]0; 0.5] is anything under water level and [0.5; 1] is above water elevation.

Considering that on earth, the highest point is Mount Everest at 8848m, when mapped to our [0; 1] range, we see it has value 0.635. So you can see the problem: most things on earth will be represented with greys really close to 0.5 and that’s why there is no contrast.

How to get nice colors and contrast?

There are several solutions, but the one proposed by the contributor was to use the “Gradient Map” plug-in. That’s a good idea. Basically you remap your greys from 0 to 1 into color gradients.
Now you can try to create a gradient by setting random stops through the GUI, but that will most likely be quite a challenge. A better idea is to do it a bit more “scientifically” (i.e. to use numbers, which you can also do through the GUI by using the new blend tool, though not as accurately as I’d like with only 2 decimal places). This is what did Massimo here by creating a gradient file which would map “magenta for invalid data, blue below zero, green to 1000 m, yellow to 2000m, and gray to white above“. From this base, I added a bit of random tweaking because I was trying to get an output similar to the NASA document (just for the sake of it), so you can get a look at how my own gradient file looks like. But if you are looking to, say, create a relief map with accurate elevation/color mapping, you’d prefer to stick by the number-only approach.

Then once you get your gradient “code”, copy it in a file with the extension .ggr inside the gradients/ folder of your GIMP config, and just use it when running “Gradient Map” filter.

Just to explain a bit the format: for each line, you get the startpoint, midpoint and endpoint coordinates (in the [0; 1] range), followed by 4 values for RGBA (also in [0; 1] range) for the startpoint then again 4 values for RGBA endpoint color. Then you get an integer for the blending mode (you likely want to keep it linear, i.e. 0, for a relief map), then the coloring value (leave it to 0 as well, which is RGB). Finally the last 2 integers are whether the startpoint and endpoint must be fixed colors, or correspond to foreground, background, etc. You will likely want to keep them as fixed colors (0).

So basically a line like this:

0.500000 0.507633 0.515267 0.000000 1.000000 0.000000 1.000000 0.000000 0.500000 0.000000 1.000000 0 0 0 0

means: gradient from 0 meter (0.5) to 1000 m ((0.515267 – 0.5) × 216 ≈ 1000) is a linear gradient from RGBA 0-1-0-1 (green) to RGBA 0-0.5-0-1. That is:

start mid end Rs Gs Bs As Re Ge Be Ae 0 0 0 0

where start is the start elevation and end the end elevation in [0; 1] range; and RsGsBsAs and ReGeBeAe are respectively the start and end gradient colors.

That’s how you can easily map the elevation into colors! I hope that’s clear! 🙂

Can’t we have nicer support with a GUI?

Yes of course. This was fun and cool to review then improve this feature, and we should not let quality patches rot in our bugtracker, but that’s not my priority (as you know) so I stopped improving the feature (if I don’t stop myself from all these funny stuff out there, when would I work on ZeMarmot?!).
I gladfully accept new patches to improve the support and have left myself 2 bug reports to leave ideas about how to improve the current tools:

  • Improve “Gradient Map” filter to provide on-canvas preview and editing, similarly to the blend tool, because I realize this filter is powerful but that is a bit of a pain to use right now (iterations of edit gradient, run the filter for test, cancel, again and again).
  • Map gradients directly from HGT import with preview and [0; 1] range remapped to elevation in meters in the dialog so that we don’t have to constantly recompute values back and forth and edit .ggr files by hand.

In the meantime, I leave this blog post so that the format is at least understandable and HGT import usable to moderately technical people. 🙂

That’s it! Hopefully this post will be useful to someone needing to process HGT files with GIMP and willing to understand how this works, until we get more intuitive support.

Reminder: my Free Software coding can be funded on
Liberapay, Patreon or Tipeee through ZeMarmot project.

Call for help: fund GIMP development and Libre animation

Too long, didn’t read? In a few words: our GIMP development + ZeMarmot production is currently funded barely above 400 € per month, this doesn’t pay the bills, my main computer broke today and Aryeom’s graphics tablet has been working badly for some time now. We are a bit bummed out.

So we call for your help!
You can fund GIMP development and ZeMarmot production on Patreon or Tipeee!

Read below for more.


If you read us regularly, you know that I am hacking GIMP a lot. We are just a handful of regular developers  in GIMP, I am one of them. My contributions go from regular bug fixes to bigger features, maintenance of several pieces of code as well as regular code review from contributed patches. I do this in the context of ZeMarmot project, with Aryeom Han, director and animator.  We draw on and hack GIMP because we believe in Free Software.
On the side, I also contribute to a lot of other Free Software.

Our absolutely-not-hidden goal is to be able, one day, to live from hacking Free Software and creating Libre Art. But clearly there is no denying that we are currently failing. With about 400€ a month for 2 people, association LILA can barely pay a few days a month (by the rules, which means a good part of the sum even goes to non-wage labour costs). These 400€ are not even the monthly rent we pay for our 1-room flat (31 m², in the far suburb of Paris); so you would assume well that we don’t live from it. We mostly live off savings and other things to pay the bills. These “other things” also use time we would rather spend on drawing and coding.

We would indeed enjoy working full-time on ZeMarmot, creating Free Software and Libre Art for everyone to enjoy. But we are very far from this point.

The main reason why we have not stopped the project already is that we promised we’d release the pilot. Funders are counting on us. Of course the other reason is that we still hope things will work out and that we will be able to live from what we love. Still the project is done at slow pace because we can’t afford to starve, right? So we are at times demoralized.

This is why I am doing this call. If you can afford it and believe that improving GIMP is important, then I would propose to fund ZeMarmot which supports paid development.
Similarly if you want to see more Libre Art, and in particular cool animation films, and maybe later other movies under Libre licenses in professional quality, then I again propose to support ZeMarmot.

» Patreon funding «
» Tipeee funding «

Our material is dying

And so why is this post released today? The situation has been hard for months now, but today it is dire: my laptop just broke. It just won’t turn on. All my data are safe since I do regular backups (and I think the hard drive is still ok anyway), but I don’t have a computer anymore to work on (I am writing this on a 8-year old 32-bit netbook which barely stands opening a few browser tabs!).

On her side, Aryeom’s graphics tablet has had issues for months. As you may remember, we partly dealt with them, but the tablet regularly shuts down for no reason, we have to remove and put back the battery or similar annoying “workarounds”. And we fear that we have to buy a new one soon.

So that’s what triggered this blog post because I realize how precarious is our situation. We barely get funding for living bills, we eat our savings and now we have (expensive) material issues. So we are calling you all who like Free Software and Libre Art. Do you believe ZeMarmot is a good thing? Do you believe our project has any meaning and that it should continue for years and years? We believe this, and have believed it for the last 2 years where we have been trying. If you do too, maybe help us a bit, relatively to your means. If you really can’t afford it, at least you can spread the word.

ZeMarmot is a wonderful experience for us, and we really don’t want it to have a bitter end (though we won’t regret a second of it).

Thanks for reading!

GIMP 2.9.6 & ZeMarmot

Note: this is a copy of a post initially posted on Patreon and Tipeee.

Splash of GIMP 2.9.6 by Aryeom
Splash of GIMP 2.9.6 by Aryeom

Last month, we released the third development version of GIMP, version 2.9.6, as preparation of the next stable version, GIMP 2.10.

Same as for previous versions, ZeMarmot project was one of the major contributors with 274 commits (out of 1885 total for this release) by Jehan, 4 by Aryeom (some icons, a new paint dynamics “Pressure Size” very useful for flat coloring, and the splash image for this development version), and even for the first time, 3 commits by Lionel, a board member of LILA association. Hence about 15% of GIMP 2.9.6 was brought to you by ZeMarmot! 🙂

To get some more insight, you can have a look at the official announcement. And if you want to get the full and accurate list of Jehan’s contributions in particular, it is available on the source repository.

Brought to you in 2.9.6 by ZeMarmot

  • made libgimp as thread-safe, which basically means simplify plug-in developer work to have plug-ins using  several cores (now all desktop computers are multi-core);
  • display angles when drawing lines;
  • code review for WebP image support, as well as some improvements and fixes (and even a patch upstream on libwebp library);
  • capability to switch exclusive visibility of layers inside layer groups only with shift-click (feature requested and tested/used by Aryeom for a few months before adding it to GIMP);
  • contributing to the Darktable and RawTherapee  developers efforts for our new “raw” plug-in allowing importing  RAW files through these third-party software and into GIMP (GIMP project advocates for cooperation with other Free Software);
  • contribution to allow GIMP to follow GEGL multi-thread limit (once again to have a better usage of modern computer processors but now in GIMP core in particular);
  • various improvements of PDF support, in particular multi-page PDF export from layers (this is the part where Lionel from LILA made his first steps as a developer with Jehan’s help!);
  • code review and fixes for improved support of PCX images import and export;
  • capacity of plug-ins to be installed in their own subdirectory,  which should in the long run allow to get rid of the “DLL hell”, in  particular on Windows system, a very common issue where some plug-ins  embed libraries breaking other plug-ins;
  • change various defaults values to get to up-to-date standards (bigger default font size, fullHD as the new default image dimension, 300 PPI default resolution instead of 72…);
  • intelligent adaptation of physical dimension precision based on printing resolution to allow better precision in various parts of the  software (measure tool, status bar, etc.);
  • capacity to choose the icon size, allowing to adapt GIMP on smaller or bigger screen and in particular high density screens, etc.;
  • auto-detection of native resolution of your screen to choose  better default icon size (this default choice can still be changed, cf.  previous point; but at least you should get better defaults);
  • vector icons by default for the various size support;
  • welcome new code contributors by adding a vim coding style file and integrating contributed emacs and kate coding style files;
  • Flatpak package for GIMP;
  • and much more! Bug fixes and minor features by the dozens!

Flatpak for creators on Linux?

For the creators who use GIMP on a GNU/Linux operating system, you may have heard of Flatpak,  the generic application package system. Since we also exclusively use  Linux, it felt important that GIMP be available in a timely manner (with  distribution package systems, it is not unheard of to have to wait  months after actual release to get some new version!). We take the opportunity of the release of 2.9.6 to test a first public Flatpak package. Since  we don’t have a stable server, we made it available to our Patreon  and Tipeee contributors only for the time being, then will try and make it available for everyone very  soon!

For information, Windows already has a GIMP  2.9.6 installer available; and a MacOS package should hopefully soon get  uploaded (it will depends on this package maintainer who has some family priorities right now). These are not maintained by us. » See the download page! « 🙂

Thanks and “en route to GIMP 2.10”!

I hope you appreciate our contributions to GIMP! Know that these are all thanks to our contributors, be them Patreon or Tipeee, in previous crowdfundings or the ones who make direct donations.

It is not easy everyday because we seriously lack funding, and we have had some blues more than once. ;-(
Yet the many of you who never failed us and continue to support us give us some courage.
Thanks to you!

We will continue in order to bring you an awesome stable GIMP 2.10. 🙂

Have fun with GIMP!

GIMP Motion: part 2 — complex animations

This is the second video to present GIMP Motion, our plug-in to create animations of professional quality in GIMP. As previously written, the code is pretty much work-in-progress, has its share of bugs and issues, and I am regularly reviewing some of the concepts as we experiment them on ZeMarmot. You are still welcome to play with the code, available on GIMP official source code repository under the same Free Software license (GPL v3 and over). Hopefully it will be at some point, not too far away, released with GIMP itself when I will deem it stable and good enough. The more funding (see in the end of the article for our crowdfunding links) we get, the faster it will happen.

Whereas the previous video was introducing “simple animations”, which are mostly animations where each layer is used as a different finale frame, this second video shows you how the plug-in handles animations where every frame can be composited from any number of layers. For instance a single layer for the background used throughout the whole animation, and separate layers for a character, other layers for a second character, and layers for other effects or objects (for instance the snow tracks in the example in the end of the video).

It also shows how we can “play” with the camera, for instance with a full cut larger than the scene where you “pan” while following the characters. In the end, we should be able to animate any effect (GEGL operations) as well. This could be to blur the background or foreground, adding light effects (lens flares for instance), or just artistic effects, even motion graphics…
All this is still very much work-in-progress.

One of the most difficult part is to find how to get the smoother experience. Rendering dozens of frames, each of these composited from several high resolution images and complex mathematical effects, takes time; yet one does not want to freeze the GUI, and the animation preview needs to be as smooth as possible as well. These are topics I worked on and experimented a lot too because these are some of the most painful aspect of working with Blender where we constantly had to render pieces of animation to see the real thing (the preview is terribly slow and we never found the right settings even with a good graphics card, 32GB of memory, a good processor, and SSD hard drives).
One of the results of my work in GIMP core should be to make libgimp finally thread-safe (my patch is still holding for review, yet it works very well for us already as you can see if you check out our branch). So it should be a very good step for all plug-ins, not only for animation only.
This allowed me to work more easily with multi-threading in my plug-in and I am pretty happy of the result so far (though I still plan a lot more work).

Another big workfield is to have a GUI as easy to use, yet powerful, as possible. We have so many issues with other software where the powerful options are just so complicated to use that we end up using them badly. That’s obviously a very difficult part (which is why it is so bad in so many software; I was not saying that’s because they are badly done: the solution is just never as easy as one can think of at first) and hopefully we will get something not too bad in the end. Aryeom is constantly reminding me and complaining of the bugs and GUI or experience issues in my software, so I have no other choices than do my best. 😉

 

You’ll note also that we work on very short animations. We actually only draw a single cut at a time in a given XCF file.  From GIMP Motion, we will then export images and will work on cut/scene transitions and other forms of compositing in another software (usually Blender VSE, but we hear a lot more good of Kdenlive lately, so we may give it a shot again; actually these 2 introduction videos were made in Kdenlive as a test). Since 2 cuts are a totally different viewpoint (per definition), there is not much interest on drawing them in the same file anyway. The other reasons is that GIMP is not made to work with thousands of high-definition layers. Even though GEGL allows GIMP to work on images bigger than memory size in theory, this may not be the best idea in practice, in particular if you want fast renders (some people tried and were not too happy, so I tested for debugging sake: that’s definitely not day-to-day workable). As long as GIMP core is made to work on images, it could be argued that it is acceptable. Maybe if animations were to make it to core one day, we could start thinking about how to be smarter on memory usage.
On the other hand, cuts are usually just a few seconds long which makes a single cut data pretty reasonable in memory. Also note that working and drawing animation films one cut at a time is a pretty standard workflow and makes complete sense (this is of course a whole different deal with live-action or 3D animation; I am really discussing the pure drawn animation style here), so this is actually not that huge of a deal for the time being.

To conclude, maybe you are wondering a bit about the term “cel animation”. Someday I guess I should explain more what was cel animation, also often called simply “traditional animation” and how our workflow is inspired by it. For now, just check Wikipedia, and you’ll see already how animation cels really fit well the concept of “layers” in GIMP. 🙂

Have a fun viewing!

ZeMarmot team

Reminder: my Free Software coding can be supported in
USD on Patreon or in EUR on Tipeee. The more we get
funding, the faster we will be able to have animation
capabilities in GIMP, along with a lot of other nice
features I work on in the same time. :-)

GIMP Motion: part 1 — basic animations

Mid-July, we finally published publicly the code of GIMP Motion, our software for animations in GIMP. It is available on GIMP official source code repository under the same Free Software license (GPL v3 and over).

We don’t have a public GIMP release containing this plugin yet. Hopefully it should happen soon, but the code is still much too experimental and incomplete. We are using it daily internally and you are welcome to do so as well, but the released version will be much better. 🙂
So it means that for the time being, if you want to play with it, you will have to build it yourself from source, or wait for someone to make a build (we may provide one at some point).

The video above describes some of the base features for simple animations, such as storyboards/animatics and most common needs for animated images (GIF, Webp…).  What we call “simple animations” is when you mostly have several images which you want to succeed at one another. No complex composition with background and character layers for instance. New features will still happen, for instance for panning/tilting/zooming on bigger panels (very common on storyboards as well), and adding various effects (a keyframed blur for instance would be a common movie effect).

We will soon publish a second part video where we will describe the more advanced features for complex animation (the ones with layered background/foreground/characters). Because we just scratched the surface of what we will be able to do with this plugin. 🙂

Have a fun viewing!

ZeMarmot team

Reminder: my Free Software coding can be supported in
USD on Patreon or in EUR on Tipeee.

Release of TinySegmenter 0.3

Today I released version 0.3 of TinySegmenter, a Japanese Tokenizer in pure Python (released in New BSD license), with a single minor fix for proper install on systems not-using UTF-8 (apparently that still exists! :P). Thanks to Mišo Belica for the patch. Apparently some of his Japanese users are using it for Sumy, his software to extract summary from texts.

About TinySegmenter and Japanese tokenization

It’s not much of a release, but it is a good occasion to tell about TinySegmenter. This is a “Tokenizer” for Japanese. What is a tokenizer? Basically it breaks sentences into words. For people who don’t know Japanese, it doesn’t use spaces or any other symbol to separate words. Theybasicallywritelikethis. Yet there are ways to break these sentences into words, usually based on statistical analysis (like most things in Natural Language Processing and Artificial Intelligence in general). For anyone who wants to know a bit more, this message from Kytea developer (another tokenizer, which is great) explains the 2 main methods with some links of software using them (among them Tinysegmenter) and especially keywords (allowing you to search more).
The reason why you want to “tokenize” Japanese or Chinese is that it is often a first step for further natural language analysis (for instance for automatic translation, grammar analysis, pronounciation hence speech synthesis, etc.).

Now the required example, “my name is Jehan” in Japanese is: 私の名前はJehanです。TinySegmenter breaks it like this:

In [7]: segmenter.tokenize(u”私の名前はJehanです。”)
Out[7]: [‘私’, ‘の’, ‘名前’, ‘は’, ‘Jehan’, ‘です’, ‘。’]

Future?

I am not planning on hacking much TinySegmenter anymore. I never was planning to; at the time I took over maintainership, I just wanted to use it for a project (which never went through) and the original developers were not answering. So I just properly packaged it, did minor changes (for instance better support of European words using Latin1 and extended Latin Unicode characters), added some tests, and that’s it. I don’t even use it anymore. Yet if more people are interested and want to use it, feel free to send me patches. I could also give commit rights, and even co-maintainership after a few patches. I just wanted to get these words out. 🙂

I also discover today the existence of a TinySegmenter3 on pypi, with less downloads than TinySegmenter (the older one I maintained, yes I know that’s a bit confusing, why would they keep the same name and just add a 3?) but worth looking at since they apparently improved performance a good deal (I haven’t checked but that’s what it says). Maybe I should look at their code and merge their commits at some points after talking to them?
We’ll see…

Anyway have fun! 🙂

Reminder: my Free Software coding can be supported in
USD on Patreon or in EUR on Tipeee.