Teammates with CI and CD

Intro

Firstly, I’d say the Teammates assignment in this semester is one of the worst coding experiences I have ever experienced. It is a bad practice for learning Google App Engine or Java EE related stuff.  Why? Have a look at these:

wow, such looooooooooooooonnnng class name, much java, very enterprise…
110 map() calls in teammates.ui.controller.ActionFactory?! 

(There are about 100 calls in the original project, we’ve added another 10. We don’t have time to refactor these stuff so we just followed they have done before.)
and that’s what I feel…

Anyway, let’s forget about the code.  This is a quick note for what I did recently. I have managed to add continuous integration and continuous delivery configurations into Teammates with Travis CI. When a commit is submitted, Travis CI will pull and compile our code, test it and deploy it to Google App Engine.

Travis CI & CI tests

The original Teammates comes with a working Travis CI configuration. We have modified a little bit to make it works.

Here is the original configuration: https://github.com/TEAMMATES/teammates/blob/master/.travis.yml

Don’t forget there are some separated unit tests and integration tests in the CI scripts and it is DIFFERENT from the local tests!!

In order to make it works for us, we need to make some changes:

  • Some original unit tests and integration tests cannot work correctly. I have fixed 600+ tests and made it works again. But there are still 29 left we couldn’t fix, which are mainly caused by frontend changes. Teammates’ tests use some hard-coded HTML files and compare the HTML generated/rendered from Teammates server. We have to write a script and crawl all the new web pages down, or manually do the job on our own, which is even more impossible. Later our tutors noticed us that we are not required to fix that as it is way too time-consuming. So we just simply disabled them as required.
  • Remove the Codecov in the Travis configuration (i.e. the bash <(curl -s https://codecov.io/bash)). It is not free for private repos, and we don’t need that according to our assignment spec. Most importantly, I’m not a crazy rich Asian and same as my friends in the group. So, we just removed it.
  • Some linting issues need to be fixed
Yay now it works…

Deployment & Continuous Delivery

On the local side, we just need to follow the instructions provided by our tutors and run appengineDeployAll Gradle task. It should work fine as we expected. But for the Travis CI side, it is a bit more complex than I thought. For some reasons, Travis CI’s GAE deployment function doesn’t work for Teammates. But we can do it by our custom script instead.

Here I assume the Travis CLI tool is installed and logged in. Follow this link for more info.

Firstly, we need to create a service account configuration from Google Cloud Console:

Download the JSON file and put in under the root of Teammate source directory

Then, pack the scripts and encrypt the archive. This requires a macOS or Linux environment as Travis CI’s CLI client cannot encrypt the file properly on Windows. According to Travis, if you are a victim of Windows, please consider upgrading to Windows 10 and use WSL. I guess msys2 or Cygwin environment should also work, but I haven’t tried it yet.

Open the terminal and do this:

~$ cd ~/wherever-your-teammate-source-is/
~$ tar Jcvf secrets.tar.xz src/main/resources/build.properties src/main/webapp/WEB-INF/appengine-web.xml gae.json
~$ travis encrypt-file secrets.tar.xz
Yes I am a bit too lazy to switch back to Ubuntu, so I’m also a victim of Windows.

After the archive is encrypted, remember to add the original files into .gitignore, especially for the service account configuration. Otherwise some days later you may get bankrupted.

Then create a Bash script for deployment, like this:

#!/bin/bash

# Decrypt config & key files
openssl aes-256-cbc -K $encrypted_*******_key -iv $encrypted_******_iv -in secrets.tar.xz.enc -out secrets.tar.xz -d
tar xvf ./secrets.tar.xz

# Clean up browser path
rm -rf firefox/

# Authorise GCloud SDK
gcloud auth activate-service-account [email protected] --key-file=gae.json --project=your-project-name

# Run Gradle deployment task
./gradlew appengineDeployAll

Now everything is done except for the Travis configuration. Just add a deploy section at the end of the configuration file:

deploy:
  provider: script
  script: bash deploy.sh
  on:
    branch: master

Here we only deploy our app for master branch, not for other buggy branches.

Teammates development notes – Initialization

Intro

In this semester there’s a course required in my program which is called Software Engineering Process & Tools. Basically it needs to form a group in the tutorial and then pick one of the four projects they’ve offered. Those four projects are some famous open source projects on GitHub, written in four different programming languages with thousands of users. We need to improve the project and add some functions according to the assignment specifications. I initially preferred choosing PHP (PhpMyAdmin) or Python (OpenPaperless) but my friends in the group prefer the Java one, which is Teammate.

Though I’ve tried writing some tiny helloworld-level Spring Boot applications with Java or Kotlin before, I haven’t looked into Java EE related stuff too much. So this is also the first time I really learn Java EE from scratch.

Here I’m using Java 8 1.8.172/1.8.181 with IntelliJ IDEA 2018.2 as my IDE on macOS 10.13.6, Windows 10 1804 and Ubuntu Budgie 18.04.1

This blog post is currently used for helping my friends in the group who may not be very familiar with Intellij IDEA.

NodeJS installation

In my opinion the best way of installing Node is using the package manager if possible. See here for more details: https://nodejs.org/en/download/package-manager/

You can also use their installation package to install if you wish. But it may be harder to do that on Linux distro (you need to set the environment variables by yourself) and you may also miss out some important updates.

Google Cloud SDK setup

Firstly we need to install Google Cloud SDK. Follow the Google’s instruction here: https://cloud.google.com/sdk

After the installation, remember to run gcloud init to log in with your Google account. Then you may need to install the Java plugin for it by running gcloud components install app-engine-java.

Importing the project

Firstly, just follow the instruction precisely from Teammate official site: https://github.com/TEAMMATES/teammates/blob/master/docs/ide-setup.md

Then when the project is imported and indexed, go to Tools -> Google Cloud Tools -> Add App Engine Support -> Google App Engine Standard:

after that, go to Run -> Run…

then “Edit Configurations”

Add a “Google App Engine Standard Local Server” by this way, shown below:

Press “OK” and it should work now.