Saturday, 16 April 2016

Graphical C++ program in the Codenvy cloud

Graphical FLTK based C++ program in the Codenvy cloud with CMAKE

Let's draw some Acid-Smileys on an FLTK window from a C++ program compiled with gcc using the CMake build tool.

You might want to read my earlier Blog about running a C++ Hello World with Codenvy if you are not familiar with Codenvy.



Step 1: Create the project by cloning this Git repo:

git@github.com:richardeigenmann/CppAcidSmileys.git

The code is based on sample code and a learning exercise from Bjarne Stroustrup's Programming -- Principles and Practice Using C++

Step 2: Create your own Runner

Expand the Runner panel by clicking on the button "Runners" at the bottom edge of the IDE window.
Click on the tab Configs. 
Then click on "+ Create New"
In the middle, change the Name to "X-Runner"
Slide the Project default slide to On
Mark all the text in the Dockerfile and remove it. Replace it with the lines below.
Click Save 

FROM codenvy/cpp_qt4
RUN sudo apt-get update
RUN sudo apt-get -y install \
   libfltk1.3-dev \
   libpng++-dev \
   libjpeg-dev \
   libxinerama-dev \
   libXft-dev \
   libfontconfig-dev \
   cmake vim
ADD $app$ /home/user/
RUN mkdir build && cd build && cmake .. && make

The FROM line is the baseline codenvy environment

The next two RUN commands request the Debian Jessie container to refresh it's repo and to install a number of development packages. Most important is libfltk1.3-dev as the program is written against this GUI framework. The other libraries are required to link the final code. They need to be installed explicitly as the libfltk package doesn't declare these as dependencies.

The ADD line copies the files from the IDE into the container

The final RUN makes a build dir, switches into it and runs cmake. Cmake looks for the FLTK library and creates a complicated Makefile. After this step all we have to do is call make to compile the program.

Step 3 execute the Runner

In the top right pick the X-Runner in the drop down menu.
Then click on the little green play icon.



Step 4 Open the VNC window

Look for the hyperlink underneath the Console panel. When it appears click on it to open the X-Window tab.



Step 5 Open a Terminal

Right click on the grey area. In the pop-up that appears click on Terminal.



Step 6 run the program

In the terminal window type: build/ClassedAcidSmiley and press Enter


(To close the GUI hit Ctrl-C on the terminal.)
(To end the runner click on the red icon in the IDE window.)

Friday, 15 April 2016

Developing and running Java Swing applications in the cloud with Codenvy

Developing and running Java Swing applications in the cloud with Codenvy

Last September I blogged about creating a C++ program on the web with the Codenvy IDE and running it in the cloud inside Docker containers. Link to blog post

This worked fine for applications that just display a text or work on the command line. But what about applications that have a Graphical User Interface? It turns out that Codenvy has a special kind of runner which includes a Virtual Network Computer (VNC) display in a browser window. All you have to do is pick the correct Runner and open a hyperlink.

So let's get started:

Create a new Codenvy Project:

  • Open https://codenvy.com
  • Click on the "New Project" button after you have signed up or logged in.
  • Pick an "Empty Project"
  • Click on the Java square
  • Inside the Java square click on the dropdown. It has Ant Project, Maven Project, Java and Google App Engine Project as options. Chose Java.
  • Give the Project a Name, for instance HelloWorldSwing
  • Describe the Project: Hello World in Java Swing
  • Then hit the "Create Project" button.




You land on the Project Page:




The open the project in the IDE:


Create a Hello.java program:

  • File > New > File 
  • Give it the name HelloWorld.java
  • Paste this code:


import javax.swing.JFrame;
import javax.swing.JLabel;
import javax.swing.SwingUtilities;
import javax.swing.border.EmptyBorder;


public class HelloWorld {
  
 private static void createAndShowGui() {
   JFrame frame = new JFrame("Hello World");
   frame.setDefaultCloseOperation( JFrame.EXIT_ON_CLOSE);
   JLabel label = new JLabel ("Hello World");
   label.setBorder(new EmptyBorder(20, 20, 20, 20));
   frame.getContentPane().add(label);
   frame.pack();
   frame.setLocationRelativeTo(null);
   frame.setVisible(true);
 }
  
  public static void main(String[] args) {
    SwingUtilities.invokeLater(new Runnable() {
        
        @Override
        public void run() {
            createAndShowGui();
        }
    });
    
  }
}

Save the file:



Choose the "VNC + Java 7" Runner:



Press the green go button:
The lower part of the screen scrolls up showing the Console, Terminal and Properties tabs.
The Docker output scrolls through the Console window and when it's ready a hyperlink appears underneath the Console scrollpanel with the heading "Application:"


Click on the hyperlink to open up the grey X Window VNC tab in your browser:


Now right-click in the grey area and choose Terminal








Compile the program with the command
javac HelloWorld.java



Run the program with the command:
java HelloWorld




Observations:

1. This terminal has apt-get installed

You can therefore install any extra package you like

sudo apt-get update
sudo apt-get install <<packagename>>

2. The run container is not linked to the IDE container 

So the source files do not update in the X-Window when you update them. You need to kill the runner and start a new runner to have changes reflected.

3. Automate your runner 

The relevant lines in the VNC + Java 7 runner are:

FROM codenvy/jdk7_vnc
RUN mkdir /home/user/app
WORKDIR /home/user/app
ADD HelloWorldSwing.zip /home/user/HelloWorldSwing.zip
RUN unzip -q /home/user/HelloWorldSwing.zip -d /home/user/app

Try the following:

FROM codenvy/jdk7_vnc
RUN mkdir /home/user/app
WORKDIR /home/user/app
ADD HelloWorldSwing.zip /home/user/HelloWorldSwing.zip
RUN unzip -q /home/user/HelloWorldSwing.zip -d /home/user/app
RUN javac HelloWorld.java

This will compile the code in the target container.
But you can't RUN java HelloWorld because while the runner is starting you don't yet have a X environment. You have to wait for the hyperlink and then open the X Window tab. There you have to open the terminal and do the java HelloWorld as shown above

4. Codenvy doesn't yet have runners for Java 8

These instructions guide you through installing Java 8 into the runner: http://tecadmin.net/install-java-8-on-debian/#
I tried them and it worked. The default java command doesn't update so you need to make sure you are picking up java 8 which is lurking in the directory /usr/lib/jvm/java-8-oracle/bin/java


Sunday, 21 February 2016

Run Cyrus IMAPD mailserver from a Docker container with the mailbox data on an external volume

The Objective:

Run an Imap server on the local Linux machine in a way that it is easy to move from one computer to the next.

The use case: I want to run the imap server on my desktop most of the time but when I go away I want to take it along with me on my laptop.

The first solution was to set up a headless virtual machine with VirtualBox and install the cyrus imap server into the virtual machine. This works and the .vdi disk images can be moved from one computer to the next. The downside is that the disk image is around 19GB in size which takes hours to copy. Also running the VM on the Laptop reduces memory and degrades performance.

The Docker solution:

Build a docker container from the latest OpenSuse image and install cyrus imapd into it. Since containers "forget" all changes when the are shut down we use a VOLUME to persist the database and mail data on the host filesystem. The host directory with the cyrus data can then by rsynced to the new machine and the container can be started there. The mail client finds the impad server on localhost:143.

The Dockerfile:

FROM opensuse:42.1

ENV mailboxuser richi
ENV mailboxpassword password

MAINTAINER Richard Eigenmann 

USER root

# add the packages needed for the cyrus server and some to work with the shell
RUN zypper --non-interactive in \
  cyrus-imapd \
  cyradm \
  cyrus-sasl-saslauthd \
  cyrus-sasl-digestmd5 \
  cyrus-sasl-crammd5 \
  sudo less \
  telnet;

# set up the saslauthd accounts (complication: the host name changes all the time!)
# -u cyrus ensures the account is set up for the hostname cyrus
# cyrus is the account we need to run the cyradm commands
RUN echo ${mailboxpassword} | saslpasswd2 -p -u cyrus -c ${mailboxuser}
RUN echo "password" | saslpasswd2 -p -u cyrus -c cyrus
RUN chgrp mail /etc/sasldb2
RUN chsh -s /bin/bash cyrus


# Set up the mailboxes by starting the cyrus imap daemon, calling up cyradm
# and running the create mailbox commands.

# Step 1: set up a sasl password valid under the build hostname (no -u param).
# Since sasl cares about the hostname the validation doesn't work on the above
# passwords with the -u cyrus hostname.

RUN echo "password" | saslpasswd2 -p -c cyrus

# Step 2: We can't use here-documents in docker so we create the instructions
# that cyradm needs to execute in a text file

RUN echo -e "createmailbox user.${mailboxuser}\ncreatemailbox user.${mailboxuser}.Archive\nexit" > /createmailbox.commands

# Step 3: Start the daemon and in the same build container run the cyradm command
# (note the ; \  at the end of the line!)

RUN /sbin/startproc -p /var/run/cyrus-master.pid /usr/lib/cyrus/bin/master -d; \
sudo -u cyrus -i cyradm --user cyrus -w password localhost < /createmailbox.commands; \
mv /createmailbox.commands /createmailbox.commands.completed;


# create a file startup.sh in the root directory
RUN echo -e "#!/bin/bash\n"\
"if [ -e /var/dostart.semaphore ]; then\n"\
"chown -R cyrus:mail /var/spool/imap /var/lib/imap\n"\
"/usr/lib/cyrus/bin/master -d\n"\
"sleep .6\n"\
"ps u --user cyrus\n"\
"fi"\
> /startup.sh; \
chmod +x /startup.sh


# start the cyrus server and a shell
CMD  /startup.sh; /bin/bash

Running the server:

Build the container:
docker build -t richi/cyrus-docker:latest .

Do these steps to set up the mail server and the host directory: Build the container:
# on the host server
mkdir /absolute/path/to/the/exported/directory/var
docker run -it --rm --hostname cyrus -v /absolute/path/to/the/exported/directory/var:/mnt richi/cyrus-docker:latest

# inside the container 
cp -r /var/* /mnt
touch /mnt/dostart.semaphore

All subsequent runs:
docker run -it --rm --hostname cyrus -p 143:143 -v /absolute/path/to/the/exported/directory/var:/var --log-driver=journald richi/cyrus-docker:latest

Testing:

telnet localhost 143

#should result in output like this:

Connected to localhost.
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE LOGINDISABLED AUTH=DIGEST-MD5 AUTH=CRAM-MD5 SASL-IR] cyrus Cyrus IMAP v2.4.18 server ready


Discussion:

Setting up the basic container and adding the cyrus software is straight forward.

Setting up the mailbox user account with the password and creating the mailbox structure is tricky: cyrus uses saslauthd to check the passwords of the users logging in. Saslauthd has some sort of anti-tamper mechanism that leverages the hostname in the validation. Since the Docker build process changes the hostname at every step this gets problematic. The saslpasswd2 -u cyrus statements set the passwords for the user account and the cyrus admin account for the hostname cyrus (the -u).

To set up a mailbox account cyrus requires the daemon to be running. The user cyrus then needs to run the cyradm command with the instructions to create the mailbox. Here documents don't seem to be supported inside Dockerfiles so we first create a script file "createmailbox.commands". We then use sudo to promote to the cyrus account and then pipe in the instructions from the script file.

This creates a Docker container that can start up and knows the user, his password and has the basic mailbox structure. You can point your mail client at this imap server and things will work fine until you restart the container. The container will forget all changes when it is shut down. Since cyrus impad stores all state in the /var directory a solution is to export the var directory to the host filesystem so that it can be easily transported to other computers as well as backed up. The -v parameter in the docker run command does just this.

The syntax of the -v parameter is the absolute (!) path of the directory on the left gets mounted to the directory on the right of the colon. Annoyingly, if you just use the -v bind-mount parameter the previous contents of the /var directory in the container are hidden and you just see the empty /var directory from the host filesystem. There doesn't appear to be a way to bind-mount the host directory so that all the obfuscated directories and files from the container "shine through" and all new writes go to the bind-mounted directory.

Therefore we must copy all the content in the container's /var to the host directory first. The way I suggest doing this is to start the container and bind-mound the host's directory to /mnt in the container. Then a cp -r can copy all content from /var to the new directory. After shutting down the container and starting it up with the directory mounted to /var we are back to the original view.

But not quite: The important directories for cyrus, /var/lib/imap and /var/spool/imap, used to be owned by cyrus:mail but are owned by root after the volume mount. Since the server feels it can't read the mailbox database if it is root owned we need to correct this before the startup. I have thus created a startup.sh script that fixed the ownership of the mounted host directory and then starts the daemon. To keep everything in one Dockerfile I create the startup stript with an echo statement right inside the Dockerfile.

To facilitate rsyncing from one host to the other I suggest chown -R user:users on the host directory. Docker runs as root and will create all new files as root owned files but can perfectly well read and write to user owned files. Userspace synchronisation tools will find it much easier to deal with user owned files, however.

Monday, 14 September 2015

Running c++ Hello World in the cloud: Codenvy and Docker

Running Hello World with the default runner

Go to Codenvy


Sign Up or Login if you have already done so.


To create the Hello World program click on "Create New Project"



Pick the "C/C++ Project", give the project the name "Hello World" and optionally a description. Click "Create"


You now have an empty workspace. Let's create the hello world program as follows:

Click on File > New > File


and give the file the name hello.cpp


Then type in the Hello World code

#include <iostream>

int main()
{
    std::cout << "Hello World!\n";
    return 0;
}



In order to make your life easier you want a Makefile that remembers all the special characters you need to type to compile and build your program. Here is a very simple Makefile:

all:
g++ -std=gnu++11 hello.cpp

# Note that Makefiles require a tab before the g++ 
# It probably doesn't copy/paste well from this blog post

Create a new file with the name Makefile and enter the above lines. Do this by clicking on the HelloWorld label next to the downward facing triangle and the click File > New > File


Now you are ready to run the Hello World program. You need to pick the C++ runner from the dropdown in the top right and then click the green Run icon.


The runner opens at the bottom of the window and shows the Console. A lot of stuff scrolls by from Docker. Codenvy uses Docker to spin up a clean Linux environment into which it loads your code. It then asks the Makefile to build your program and executes your compiled program afterwards. You can see the output towards the bottom of the Console output.


Interactive shell

That was fun but can we have an interactive shell? How can we make the program prompt the user for her name and play it back? We need a different runner!

Click at the bottom of the screen on "Runners" and then pick "Configs" on the left side.



Then pick + Create New and the Properties dialog opens up for the new runner with the disingenuous name "ENV1-HelloWorld". Rename it to Interactive.


The green text is the Docker file that Codeenvy will use to start up the Docker runtime. The available commands are documented on the Docker website. Replace all the lines in the Dockerfile with these commands:

FROM codenvy/shellinabox
RUN sudo apt-get update && sudo apt-get -y install g++
ENV CODENVY_APP_BIND_DIR /home/user/app
CMD sleep 365d

Then click save on the right hand side.


I had to untick and tick the "Matches project" to make it show up in the left list.

Also make this the default Runner for the Project with the sliding button in the bottom middle.


Now run the project with the new "Interactive" runner. Pick the Interactive runner in the top right corner and click the green run icon. The Console scrolls through...


Now click on "Terminal". After a moment you are connected to the terminal of the Docker environment that was spun up. You can now use standard Linux shell commands to compile your program and run it. Try these commands:

cd app
make
./a.out


Now let's make our program interactive by asking for the name and playing it back. 
Go to the hello.cpp program and change the code to read as follows:

#include <iostream>
#include <string>

int main()
{
   std::cout << "Your name please: "; 
   std::string name;
   std::cin >> name;
   std::cout << "Hello: " << name << "!\n";

   return 0; 
}



If you kept the terminal session open you don't need to do cd app. Then enter the following commands:

make
./a.out

The shell will rebuild the changed program and the ./a.out will make it run, asking for your name and playing it back.

When you are done with the Docker runtime close it by clicking the red button.






Sunday, 9 March 2014

Don't call it "new"! But calling it "old" is OK.

Have you ever seen a directory with files names like this?














Which is the current one and which are older working copies?

The super organised person would name the files like this:














For the rest, let me recommend calling the files "old" as you have no trouble picking out the latest version here:















Of course that means you need to save the changes in 2 steps. You will have to save the new version of the document under a temporary name close the document, go to the explorer and rename 2 files. The effort is worth it!

Monday, 1 April 2013

Best advice I ever got

The best advice I ever got came from Renée Watkins. She gave me hard time over some software I had written to book FX trading. She kept asking me for detail upon detail and I just didn't know all the answers. Eventually she recommended that I ask WHY?  It took this to heart and it has helped me no end! If you don't know why something is supposed to work this way or that then whatever you code will not fit the expectations of your user.

It also ties in with another favourite from work: "No surprises ". If you ask enough probing questions then you will understand the problem being solved and will avoid many unpleasant surprises.

Backups

OK, you say, I get it, we should back up our data! And then you make a half hearted attempt and move on. But deep down you know about MTBF, the Mean Time Between Failures. The one thing we can say for certain about mechanical systems (such as your Hard Disk) is: IT WILL FAIL. The MTBF might give you the confidence that "my hard disk is likely to go on for another 3 years" and I sure hope it does. And when it does fail, often its not completely dead and you can get much of your data off it...

So my suggestion is to keep your data fully replicated on multiple devices. In order to do this easily I find it best to have one single directory underneath which everything of importance goes. (Think about it: When your disk blows in 3 years your computer is old and you will replace it with the shiniest new one that your budget allows. It will have a new version of Windows on it with new versions of the applications you use (and icons all looking different and in unusual places.) You don't need a backup of the Operating System and the Programs; you just need a backup of your data. [Yes, a list of the programs you use will be helpful! Perhaps you should go off and create just such a list in Evernote right now?]

I suggest you have one directory on the root of the filesystem (say c:\) with the name of the person. Example:  c:\Tom

Then you need to consider what kind of data you have. Some of this will be insensitive such as eBooks, mp3s, movies whilst you may feel other data is somewhat more private in nature like your salary slips, tax filings, accounts or contracts. You can grant and revoke permissions at a directory level so I suggest you create the insensitive directories directly under the main directory and create a Private directory for the more sensitive stuff. I.e.:

c:\Tom\Mp3
c:\Tom\Pictures
c:\Tom\Movies
c:\Tom\ToDo
c:\Tom\Private
c:\Tom\Private\Taxes
c:\Tom\Private\Contracts
c:\Tom\Private\Contracts\HealthInsurance
c:\Tom\Private\Accounts

You need to decide where the pictures should go. You probably want to share them with friends and family so they would more likely go into the main directory than the "Private" directory. (If aunt Mathilda is sitting next to you do you really want to be clicking around in the "Private" directory?)

I find it very useful to have a "ToDo" folder. This is supposed to be empty but will take all temporary stuff that you haven't filed properly yet.

The goal is to have all your data somewhere under c:\Tom and nothing on your Desktop, nothing in c:\documents and settings\local user\My Pictures\ and other crazy locations. There is an added advantage to this because some programmers seem to think that they can freely create junk files in your "Documents and Settings" folder. You have no idea what these files are and don't know if you can delete them without breaking anything. By having your data in your own structure they can freely use those locations and you will just walk away from that pile of junk when you upgrade to your next computer.

Now you are ready to do something about your backups! In the simplest form you just copy the entire c:\Tom folder to an external Hard Disk. Buy a large one and call the copy something like "\Tom-Backup-2013-03-01" and the next one "\Tom-Backup-2013-04-01". This allows you to go back to an old backup if you discover a file was corrupted or you accidentally lost half the text of your thesis some time in March.

I own multiple computers and like to have the whole directory replicated to each machine (in the belief that not all disks will fail at the same time). The problem you get into is that between copies different files will be modified on each of the machines. You need clever software to figure out what files were modified on which machine so that the latest version can be copied over. My favourite software for this is Unison File Synchronizer. It works really fast on huge directories between two Linux machines and works well (but slower) when comparing two directories (one local, one remote) on Windows.

For backups I recommend Box Backup. This backup software looks for changes on the filesystem and encrypts the changes and uploads them to the backup server. By searching for the changes it doesn't have to upload all 100MB of the file, just the parts that actually changed. Because it stores the changes it can reconstruct a file from several changes back. Because it encrypts the data on the client the person running the server can't decrypt the data. It runs in the background and figures out what to do completely on it's own. The client comes for Linux and Windows whilst the server needs to run Linux. The downside is that it is difficult to set up (especially the bit with the cryptography keys). Also most home users are throttled on the uplink of their internet connection which makes backups very slow. At worst the Internet will seem slow because the page requests have to queue up behind large backup packets.

Update on 19 May 2013: http://freefilesync.sourceforge.net/ looks like an interesting alternative to Unison for directory synchronisation.