Parallel processing batches of images with ImageMagick, reconsidered

02 05 2011

Recently I showed some bash scripts to process images with ImageMagick in parallel. Boy, was I astonished when I read more about GNUs "parallel" command: It does not only scale perfectly on all CPUs and all cores, it also can use several machines via the network.

The task to scale down all photographs can be condensed into this simple command line:

1024$ ls ../all/*.JPG | parallel convert {} -resize 1024x1024 {/.}_preview.jpg

See also the great comments to this blog entry about "parallel".

Parallel processing batches of images with ImageMagick

29 04 2011

From my recent holiday trip to Lanzarote I brought back nearly 600 photographs. When I reviewed them, it became quite clear that my picture-taking skills are not that impressive and most of the images, if not all, need some kind of retouching, most notably, adjusting the white-balance.

The Gimp becomes more and more my friend, but processing all of the photographs manually would rather quickly wear out my mouse arm and index finger. Some kind of scripting would be nice here, as scripts also can be kept for later reference on what I did.

Continue reading "Parallel processing batches of images with ImageMagick"

Finding stuff in multiple files and opening them in gVIM

21 02 2010

Currently I'm working on some DokuWiki template which is based on the arctic template. DokuWiki itself scatters the CSS styles in several files like 'layout.css', 'design.css etc. The arctic template adds its own bunch of files like 'arctic_layout.css', 'arctic_design.css' and so on.

Now I needed to change the color of some page element. But where in this plethora of style files is the matching CSS definition? I called upon several good friends to find it.

Firstly I fired up FireBug which nicely showed which CSS classes were applied to that page element. As it turned out, the element was wrapped in a div tag and formatted by a CSS class named 'meta'.

Fine. Now I need to find the string "div.meta" (or, maybe just ".meta") in the style files. Of course we know the commands 'find' and 'grep' will quickly show us the results, but today -- as an additional sugar -- I wanted to open all found files together in the same instance of gVIM. Ah this is why I love the command line:

find . -name '*.css' -or -name '*.php' | xargs grep div.meta \
|awk -F: '{print $1;}' |sort -u | xargs gvim

Let's split that command up:

  1. The find command lets us only look into the wanted files. And it does that recursively, if we would need that.
  2. With xargs we feed those filenames nicely into grep. And grep prints each line which matches the expression, prepended with the name of the file where the line was found.
  3. We are only interested in the filenames, so awk will get them for us: "split the line at colons and print the first column".
  4. Maybe there are more than one matching lines in the same file. Therefore we build an unique list of 'em.
  5. And lastly, let xargs provide the list as arguments to gVIM

That's it.

Well, not quite. :-) DokuWiki allows to use some kind of variables in the CSS files. Consequently, the file which defines the CSS for 'div.meta' does not contain the sought color information, but the name of a variable. And this, FINALLY, I can change in the file 'style.ini'. Argh.

Repair grub with grml

14 10 2009
My development box runs a bleeding edge Debian sid installation. One day an update produced the message that grub 0.97 is now called grub-legacy and that it will chainload into the new (extremely bleeding edge) grub 2. Should that work I could run upgrade-from-grub-legacy. Well, that sounded nice and I researched what is the status of Grub 2. Almost everybody who wrote about Grub 2 noted, that documentation lacks a lot -- so I won't reproduce this statement here ;-) Fortunately, in the caveat section the draft of the Grub 2 manual gives the following hint
deb upgrades - In the Debian 5.x and Ubuntu 9.x series, upgrading to GRUB 2 after an an LVM/LVM+crypto installation (under GRUB legacy) will leave you unhappy.
Hmpf. Made a mental note not to touch above command, because I am using lvm. But yesterday evening my mental state was kind of unconscious and out of -- whatever it was, that had driven me -- I entered upgrade-from-grub-legacy. A dialogue asked me which devices Grub 2 shall be installed on and -- @$""§ again without thinking -- I selected both drives. [Deviation] The computer has two different brands of hard drives drives, an old one and a new one, with different sizes and partitioning. The system used to identify them as /dev/hda and /dev/sda. A recent upgrade of the kernel introduced some confusion here, because suddenly the hda was populated as sdb! Researching (many thanks to all the people who write about their knowledge) found that the new kernels see all drives as "sd". So I changed the fstab and everything worked again. [/Deviation] Back to Grub 2. The installation and a reboot finished without errors. Null sweat here. But I noticed that the Vista entry in Grub's menu was missing. So I installed and run os-prober which found the Vista partition and also produced an entry for the splash screen (which I disliked but accepted). Strangely, it did not list my Linux kernels! As Whitesnake had put it:
When I first saw you baby You took my breath away, I knew your name was Trouble But, my heart got in the way I couldn't stop myself from reaching out, I could not turn away
In my case my brain just passed out. I did a reboot. Great, the Grub menu appeared and an entry for Vista was there. -- BUT IT WAS THE ONLY ONE! Uniqueness might be applicable for certain rings, but certainly it was displeasing for my boot menu. Very! Hardened from battling with overwritten Lilo boot records in the past, and in the very comforting situation of having a quite recent backup I did not break into sweat. Browsing through the pile of stuff on my desk produced two things, in that order: a disc of grml Schluchtenscheisser, and a grin on my face.

Linux Virus Shock?

23 06 2009
A couple of minutes ago something weird happened on my Linux box: I had been working on some graphics in Gimp and was clicking through the windows when suddenly a song started to play. I was startled, and it lasted only for a few notes and then fell silent again. My first (paranoid) suspicion was, somehow a virus had pirated my machine or, some hacker was trying out commands in a shell... Hmm, a less into /var/log/auth.log showed nothing suspicious. But then, a capable hacker could have concealed her traces. Unfortunately I had no more ideas about how to find out the source of the song and so I finished the graphics. When the desktop reappeared after all of Gimp's windows were closed, I remembered that I had saved that song onto the desktop. And there was another mp3. I wanted to listen to it again and moved the mouse cursor upon its icon, but -- before I could double-click, the song started to play! Aww, I was tricked by the (new?) Gnome (or Nautilus?) behaviour, to play a sound file immediately when the cursor hovers it. That must have happened when I browsed through Gimp's windows earlier on!

Today's favourite documentation egg

22 02 2009
:nunmap can also be used outside of a monastery.
Found in VIM's help, describing the various map commands.

Microsoft Outlook Security -- WTF?

16 12 2008
Aus aktuellem Anlass suche ich im Internet nach Informationen ueber die Sicherheit der Emailanzeige in Microsoft Outlook. Obwohl ich schon einiges gewoehnt bin von Microsoft, verschlaegt es mir bei folgender Dreistigkeit doch dermassen die Sprache, dass ich entgegen meiner sonstigen Gewohnheit nicht einmal mehr faehig bin, in Englisch zu schreiben. Doch der Reihe nach. Zunaechst schaute ich bei Google, welche Seiten zum Thema "Outlook Sicherheit" zu finden sind. Suchergebnisse Prima, es sind sogar 2 Links zu Artikeln bei Microsoft mit unter den ersten Ergebnissen: 1 und 2. Aber statt zu den erhofften Artikeln fuehren sie lediglich zu dem Hinweis "Um die optimale Anzeige Ihrer Webseiten zu unterstuetzen, wird ueberprueft, ob ein Programm von 2007 Microsoft Office installiert ist." Wow. Dass man neuerdings zur Ansicht von Webseiten nicht mehr einen Browser wie Firefox oder auch Microsofts Internet Explorer benoetigt, sondern ein Microsoft Office der neuesten Generation, ist allerdings ein starkes Stueck. screen02 Natuerlich surfe ich mit abgeschaltetem JavaScript, das hat jene Seite richtig erkannt, und so klicke ich selbst auf den angebotenen Link, endlich zum erwarteten Artikel zu gelangen. Aber weit gefehlt! Die folgende Seite bestaetigt, -- nun auf bunte Weise --, dass die Informationen nur fuer Besitzer von MS Office zugaenglich sind, und man wird gleich aufgefordert, die neueste Version zu kaufen. Ist doch interessant, dass man jedoch diese Webseite ohne JavaScript und auch gänzlich ohne Microsoft-Programme lesen kann! (Hervorhebung im Bild von mir.) Was haben wir gelernt?
  1. Die Aussage "Um die optimale Anzeige Ihrer Webseiten zu unterstuetzen", dient nur dazu, einen Laien zum Kauf von MS Office zu noetigen. Was ist an der Anzeige der bunten Webseite nicht optimal, ganz ohne Microsoft Programme? -- Abgesehen davon, dass sie nicht den versprochenen Inhalt hat.
  2. Informationen zu Office bekommt man von Microsoft offenbar nur, wenn man schon Office gekauft hat. Sich vorab zu informieren oder als unabhaengiger Consultant sich zu informieren, ist ausgeschlossen.
  3. Nicht nur wird man zum Kauf von MS Office genoetigt, man muss auch Microsoft erlauben, den eigenen Computer zu durchsuchen! Unser Bundestrojaner wird vor Neid erblassen!
So, WTF? Eigentlich wollte ich mich nur ueber MS Outlook informieren, um meine Kunden gut unterstuetzen zu koennen, aber was passiert? 2 Stunden verschwendet fuer nichts. Danke, Microsoft.


25 08 2008
The long planned documentation for my web application framework Parenchym is online. Finally I found a way to describe and present it that suits me right. Since the pages have just started, most of them are in German, but translation into English is under way.

My new photoalbum is online

03 08 2008
I am happy to announce that Pixy, my new photoalbum is now online. Visit Pixy here

Convert your photo collections into an online album

02 08 2008
It has been a long time since I lastly uploaded a new gallery to DigiPics, my old gallery in The Annwn. Now, this does not mean, I had not taken any pictures in the meantime. Au contraire, mon capitain! But how could I handle all those photographs more efficiently? For some picture sets I had made special pages to display them, as you can see in DigiPics. But I only want to repeat this for special occasions. For now I want a method to process the images, like converting them to suitable sizes for displaying on screen, and show them in some decent album, where the visitior can browse through sets and all of that. Additionally, the EXIF data of the photographs should be displayed, so that one could get an impression how they were taken. Of course, you could argue, there already are plenty of picture sharing sites. Yeah, what about being techie if I'd use someone else's program? ;-) What I finally have produced are (1) some scripts to process the images, and (2) a small application to display the albums. First, the scripts All scripts (two, in fact) can be installed into Gnome's file browser Nautilus, so you only have to select the images to process, right-click and select the script from the context menu. create_thumbs is a small shell script that resamples the selected images into various sizes, 120px for thumbnails, 640px and 1024px. It also rotates them according to the orientation flag in the EXIF data. Of course, the script relies on external programs to accomplish this, namely convert from ImageMagick, jhead and jpegtran. get_exif is a Perl script that fetches all EXIF data from the original image and stores in in a separate file. I use Image::ExifTool for this. In a first attempt I used PHP's EXIF functions, but as it turned out, the Perl lib is better, especially because it interprets the MakerNotes quite nicely. Download the scripts Second, the application With the scripts we produced a nice gallery and now we need some way to display it. Pixy does right this. It is (almost) a single-file PHP script, which borrows some code and ideas from my Web Application Framework Parenchym. Look inside pixy.php for more information how to set the galleries up. In the next days I will upload my galleries, so that you can see Pixy live. Maybe I'll also publish the sources.

PHP Madness: Floats, type casts and locales

26 07 2007
While working on my framework Parenchym, I encountered the following situation: In a form a user has to input a floating point value, which will then get processed. After that, the form with the latest input is re-displayed. Now, Parenchym is very thorough when it comes to checking the user's input. Not only does it check whether the value is valid or not, it also casts PHP's internal type to the right one. As you will remember, all input is recieved by PHP as string. So, the user's float comes in as a string as well, but Parechym converts that to float again. So far, so good. Enter the user's locale. For this examination, we'll suppose it is set to `de_DE.utf8'. The first input poses no problem, because the user (which was me, at that time) is used to input floats with a decimal point. The input of 8.3 enters the script as "8.3" and Parenchym returns it as 8.3. Great. But the shit hits the fan when the form, and so the float, is displayed again. This time the locale settings strike! The value in the form field is suddenly "8,3". Mind the decimal comma here! Submitted again, PHP's type cast -- completely ignorant of the locale settings -- truncates the float, rendering it as 8! WTF! Lesson learned today: PHP respects locale settings when printing floats, it DOES NOT respect locale settings when doing type casts!

NVidia Driver 1.0-8756 on Xorg 7

28 04 2006
Currently, Debian's package list does contain the most recent version of NVidia-GLX, but not the required NVidia-kernel package with the same version. Well, alternatively, we could tell Xorg to use its own nv driver. -- If we do not need 3D speed, and can work without dual head graphics cards. NVidia Corp. provides a decent installer for Linux, which will help us here. Download it from their website and run it, as the provided README tells. The installer did not find any kernel interface matching my 2.6.16, so I installed the matching kernel headers and adviced the installer to build them. This step works well. But after rebooting my machine, Xorg refuses to start, complaining it cannot find the nvidia driver. A look into the error output reveals, Xorg looks for its drivers in some other place than good old X11R6 did. NVidia's installer apparently had not noticed that and stored the modules in the place for X11R6. Perform this on the command line, and everything should be settled:
 cp /usr/X11R6/lib/modules/drivers/nvidia_drv.* /usr/lib/xorg/modules/drivers/ 

Connect Nokia N70 with gnokii

21 04 2006
Some time ago I bought a Nokia N70 and, being absent from Windows several years now, I wanted to connect it with my Linux box running Debian. A web search produced several tools which promised to connect to mobile phones via bluetooth. gnokii proved the most versatile. After some trial and error, gnokii talked to my phone and was able to read out the contents of the addressbook. I used version 0.6.11 of gnokii then, and it turned out that the N70 was able to store more fields in the addressbook than gnokii dumped. So I asked about that matter in the mailing list. Alas, always being short of time, I didn't follow the proceedings thoroughly, so I cannot say if this issue is solved by now. Since then, I recieved several mails, asking about the procedure and the settings to establish a fruitful discourse between the N70 and gnokii. This is what I did: 1. Compile and install gnokii Compilation (of version 0.6.11) was flawless. Just the usual configure, make, make install. No special switches for configure needed. 2. Upload gnapplet to the N70persoenlich_gnapplet.jpg Ahem, well *blush* Ok, now and then a Windows machine comes in handy... On a windows machine I installed the Nokia software which came with the phone, but it did not work with the Windows bluetooth driver. Downloading the latest version from Nokia's web site did the trick. From here, I uploaded gnapplet to the phone. You can find it in the folder "Perönlich" (maybe 'personal' on English phones), where the games are. 3. Pair the computer and the N70 via bluetooth Back to Linux now... While the bluetooth on the computer is active, also activate bluetooth on the N70 and make it visible to others. bluetooth_config.jpgbluetooth_devices.jpgUsing the Debian packages for bluetooth on a ThinkPad T30, only minor configuration settings are necessary. Here are the hcid.conf and rfcomm.conf. I'm not sure, if the rfcomm settings are necessary at this point, since their values I obtained after a successful connection to the N70 (see below). Also I created /etc/bluetooth/pin, which is a plain text file with only the numeric PIN in it. For detailed and complete (incl. Kernel config etc.) instructions, read the Gentoo Linux Bluetooth Guide. 4. Configure gnokii gnokii's config file is /etc/gnokiirc. Set at least the following fields: port Since we are connecting via bluetooth, this is the MAC address of the phone. We can obtain it by issuing the following command, which also lists the phone's name: hcitool scan model Model is series60. connection Connection is bluetooth. rfcomm_channelgnapplet_started.jpg Start gnapplet on the N70. It will print out several informations on the display. Among them, the here needed channel no: RfCOMM: on, Ch#14. Here is the complete gnokiirc. 5. Use gnokii Start gnapplet on the N70 (it will surely be timed out by now). Issue gnokii commands, e.g. gnokii --identify Good luck.

Howto reconfigure the keyboard

14 04 2006
After updating my Debian box to Xorg 7 and rebooting, a nasty keyboard error appeared in Gnome. And several keys did not work anymore, the function keys, the ALT-GR etc. As a user of a German keyboard layout, this left me without the possibility to switch to a text console, without the `@', without square and curly braces, etc. Fortunately, Firefox does not need those special keys and I looked in the web for help. Which these pages provided fairly well: The overall solution to avoid the error message is, afaik, to manually set the keyboard model in X to the same value as in Gnome:
$ xprop -root | grep XKB
_XKB_RULES_NAMES_BACKUP(STRING) = "xorg", "microsoftpro", "de", "nodeadkeys", ""
_XKB_RULES_NAMES(STRING) = "xorg", "microsoftpro", "de", "nodeadkeys", ""

$ gconftool-2 -R /desktop/gnome/peripherals/keyboard/kbd
layouts = []
model = microsoftpro
overrideSettings = false
options = [eurosign	eurosign:e]
Here's the full section in xorg.conf:
Section "InputDevice"
Identifier     "Generic Keyboard"
#Driver         "keyboard"
Driver         "kbd"
Option         "CoreKeyboard"
#Option         "XkbRules" "xfree86"
Option         "XkbRules" "xorg"
Option         "XkbModel" "microsoftpro"
#Option         "XkbModel" "pc105"
#Option         "XkbModel" "pc104"
Option         "XkbLayout" "de"
Option         "XkbVariant" "nodeadkeys"
The old setting said the model is `pc105', but this did not work anymore. Then I tried `pc104' both in X and in Gnome and at first it looked good. But left me wondering, which key the 105th key is... Well, it is the key with the greater-than/lesser-than symbols and the pipe. `pc104' rendered it out of order. Blech! But the Gnome keyboard dialog lists a M$ Natural Keyboard, and setting this one in Gnome and X: < << >>> ||| OH YES! :-) Belegungenbelegungen_hinzu.png Hmm, what about the Euro-key? Switch it on here: belegungseinstellungen.png And lastly, there's the neat feature to force me having a short break every hour or so: tipppause.png This was the error message: kbd_fehler.png

Parenchym (upd)

17 08 2005

Es gibt jetzt eine kurze Einleitung zu Parenchym - woher z.B. der komische Name kommt. Und die erste Dokumentation zu den Klassen ist hochgeladen: siehe Parenchym/lib/SamilFactory.

Der Artikel beschäftigt sich außerdem ausführlich damit, wie man in PHP viele Klassen verwalten und automatisch on demand laden kann. Ohne endlose require_once-Orgien.

Use the menu below to choose a header image. You can choose between images included with the theme or use a custom image.

If you are using both sidebars and you want to use an included image then choose one ending with "_large.jpg". For ex: red_sky_large.jpg instead of red_sky.jpg

To choose a custom image, select "custom header path" from the menu list and then click on the "media library" link. A custom header image must be 780x95 in size if you are using only one sidebar or 1000x95 if you are using both sidebars.

Please choose a header image: OUT ); define("FRESHY_CUSTOM_HEADER","Custom header path:"); define("FRESHY_HOMELINK_DESCRIPTION","Enter a custom label for the link pointing to your blog front page: "); define("FRESHY_NAVLINK_TITLE","Navigation link "); define("FRESHY_NAVLINK_DESCRIPTION_URL","Enter URL for Navigation link "); define("FRESHY_NAVLINK_ANCHOR_TEXT","Enter label for Navigation link "); define("FRESHY_NAVBG_TITLE","Navigation highlight colour"); define("FRESHY_NAVBG_DESCRIPTION","Please choose a navigation button highlight colour for this theme: "); define("FRESHY_NAVBG_GREEN","Green"); define("FRESHY_NAVBG_RED","Red"); define("FRESHY_NAVBG_LIGHT_BLUE","Light Blue"); define("FRESHY_NAVBG_BLUE","Blue"); define("FRESHY_NAVBG_PURPLE","Purple"); define("FRESHY_NAVBG_GRAY","Gray"); define("FRESHY_NAVBG_ORANGE","Orange"); define("FRESHY_NAVLINK_HOME","Home"); //only used for s9y versions <1.1 ?>