running simple Java HTTP server on OSv and EC2


, , , , , ,

Be able to run Java App on top of EC2 with OSv ( as minimalistic operating system as possible ) . Create AMI with custom jar and start application once machine is up.

Using Capstan should be way to go, create new image, test it on QEMU or Virtual Box and once ready produce EC2 image. That path didn’t go well for me, I had issues installing Capstan, running sample image initially on Mac and later trying on EC2.

Instead I went in different direction, used one of already available OSv AMIs. I used OSv-Tomcat to ensure I have Java ready. OSv comes with Java 1.7. EC2 instance used was m3.medium. I was unable to get dashboard running on m4.medium. Easiest way to ensure instance is up and ready is to check Dashboard running on 8000. Make sure port 8000 is allowed in Security Group ( for my HTTP server I also used additional 7071 port ). Tomcat which is started out of box is running on 8081.

Next step is customization. For all required steps Dashboard REST API was enough. Create uber jar of custom http server, I was using simple-http-server given in links, running mvn clean package. Upload uber jar over Dashboard REST File API.

Use Dashboard REST OS API to change startup command, original command was io.osv.MultiJarLoader -mains /etc/javamains

to ( http server uber jar is named app.jar on OSv FS ) -jar app.jar

Use Dashboard REST OS API to restart OS. Once OS is restarted custom http server should be available on 7071 port.


should return

Hello World on: /test.


Create AMI from EC2 EBS volume and use it to spin tons of simple http servers 🙂








chunked-seq, sequence and mapcat


, , , ,

    (fn [element] (println "realizing" element) element) 
    (range 0 100)))
> realizing 0
> realizing 1
> realizing 2
> realizing 30
> realizing 31
=> 0


    (fn [element] (println "realizing" element) element) 
> realizing 0
=> 0


    (fn [element] (println "realizing" element) element) 
    '(0 1 2 3 4)))
> realizing 0
=> 0


(chunked-seq? (range 0 4))
=> true
(chunked-seq? '(0 1 2 3))
=> false
(chunked-seq? (range))
=> false

mapcat, transducers, sequence:

    (mapcat (fn [element] (println "realizing" element) [element]))
> realizing 0
> realizing 1
> realizing 2
> realizing 31
> realizing 32
=> (0 1)


    (mapcat (fn [element] (println "realizing" element) [element])) 
    (take 2)) 
> realizing 0
> realizing 1
=> (0 1)


    (fn [element] (println "realizing" element) [element])
> realizing 0
> realizing 1
> realizing 2
> realizing 3
=> (0 1)

array -> dictionary with key fn


, , ,

class Utils {
    class func indexSequence<K, V>(
        _ sequence: [V],
        _ keyComputeFn: (_ value: V) -> K) -> Dictionary<K, V> {
        return sequence.reduce(Dictionary<K,V>()) { (result, value) in
            var result = result
            result[keyComputeFn(value)] = value
            return result


let sequence = ["a", "ab", "abc"]
let wordCountFn: (String) -> Int = { word in word.count }

let result = Utils.indexSequence(sequence, wordCountFn)
// [2: "ab", 3: "abc", 1: "a"]

Clojure type hinting


, , , , , ,

Clojure code snipet

(let [headers (vector "A" "B" "C")]
  (println "with type hint")
  (.indexOf ^java.util.List headers "B")
  (println "without type hint")
  (.indexOf headers "B"))

generates following byte code

 0: getstatic     #15                 // Field const__0:Lclojure/lang/Var;
 3: invokevirtual #20                 // Method clojure/lang/Var.getRawRoot:()Ljava/lang/Object;
 6: checkcast     #22                 // class clojure/lang/IFn
 9: ldc           #24                 // String A
11: ldc           #26                 // String B
13: ldc           #28                 // String C
15: invokeinterface #32,  4           // InterfaceMethod clojure/lang/IFn.invoke:(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
20: astore_0
21: getstatic     #35                 // Field const__1:Lclojure/lang/Var;
24: invokevirtual #20                 // Method clojure/lang/Var.getRawRoot:()Ljava/lang/Object;
27: checkcast     #22                 // class clojure/lang/IFn
30: ldc           #37                 // String with type hint
32: invokeinterface #40,  2           // InterfaceMethod clojure/lang/IFn.invoke:(Ljava/lang/Object;)Ljava/lang/Object;
37: pop
38: aload_0
39: checkcast     #42                 // class java/util/List
42: ldc           #26                 // String B
44: invokeinterface #46,  2           // InterfaceMethod java/util/List.indexOf:(Ljava/lang/Object;)I
49: invokestatic  #52                 // Method java/lang/Integer.valueOf:(I)Ljava/lang/Integer;
52: pop
53: getstatic     #35                 // Field const__1:Lclojure/lang/Var;
56: invokevirtual #20                 // Method clojure/lang/Var.getRawRoot:()Ljava/lang/Object;
59: checkcast     #22                 // class clojure/lang/IFn
62: ldc           #54                 // String without type hint
64: invokeinterface #40,  2           // InterfaceMethod clojure/lang/IFn.invoke:(Ljava/lang/Object;)Ljava/lang/Object;
69: pop
70: aload_0
71: aconst_null
72: astore_0
73: ldc           #55                 // String indexOf
75: iconst_1
76: anewarray     #57                 // class java/lang/Object
79: dup
80: iconst_0
81: ldc           #26                 // String B
83: aastore
84: invokestatic  #63                 // Method clojure/lang/Reflector.invokeInstanceMethod:(Ljava/lang/Object;Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/Object;
87: areturn

focus should be on

38: aload_0
39: checkcast     #42                 // class java/util/List
42: ldc           #26                 // String B
44: invokeinterface #46,  2           // InterfaceMethod java/util/List.indexOf:(Ljava/lang/Object;)I
49: invokestatic  #52                 // Method java/lang/Integer.valueOf:(I)Ljava/lang/Integer;


70: aload_0
71: aconst_null
72: astore_0
73: ldc           #55                 // String indexOf
75: iconst_1
76: anewarray     #57                 // class java/lang/Object
79: dup
80: iconst_0
81: ldc           #26                 // String B
83: aastore
84: invokestatic  #63                 // Method clojure/lang/Reflector.invokeInstanceMethod:(Ljava/lang/Object;Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/Object;

One of ways to generate given sample is to create new lein project, define test namespace and add given clojure snipet. Add namespace to :aot in project.clj. Perform lein jar, unpack jar, and use javap -private -c -verbose on one of namespace generated class files …

drawing the world map


, , , , , ,


prepare dataset
download and extract to /tmp/dataset/, you should have ne_110m_geography_regions_polys directory inside containing ne_110m_geography_regions_polys.shp and other files.

checkout code and run repl

git clone
cd clj-geo
lein repl


create continents image

(require 'clj-geo.visualization.raster)
(require 'clj-geo.env)
(require 'clj-common.2d)
(require 'clj-common.localfs)
(binding [clj-geo.env/*dataset-path* ["tmp" "dataset"]]
  (let [continent-image (clj-geo.visualization.raster/create-continent-image)]
      (clj-common.localfs/output-stream ["tmp" "continents.bmp"]))))

multiple-reduce, clojure macro


, , , , ,

(defmacro multiple-reduce
  [bindings coll]
     (fn [state# value#]
             (fn [[key# default# func#]]
               [key# (func# (get state# key# default#) value#)])

Macro for doing multiple parallel aggregations on objects sequence resulting in single aggregate object.

multiple-reduce is defined under clj-common project


    :inc 0 #(+ %1 (inc %2))
    :dec 0 #(+ %1 (dec %2))]
  '(1 2 3))

will result with

{:inc 9, :dec 3}

Each binding is defined with  keyword that will contain aggregated value after aggregation is finished, initial value and aggregate function ( f(state, value) -> state ). Aggregate function is function of two args, aggregated value ( state ) and increment value ( value ) which with each iteration returns new state.

Example usage from my project

(let [first-location (first locations-seq)
      aggregate (multiple-reduce
                    :max-longitude (:longitude first-location) (partial max-aggregate :longitude)
                    :min-longitude (:longitude first-location) (partial min-aggregate :latitude)
                    :max-latitude (:latitude first-location) (partial max-aggregate :latitude)
                    :min-latitude (:latitude first-location) (partial min-aggregate :latitude)
                    :distance [0 first-location] distance-aggregate
                    :last-location first-location last-location-aggregate
                    :activity-duration [0 first-location] duration-without-pause]
  ;continue computation on aggregate


Where distance-aggregate is defined as

(defn distance-aggregate [[distance previous-location] location]
  [(+ distance (geo/distance-two-locations previous-location location)) location])

CloudKit as exchange between iOS app and JVM backend service


With CloudKit Web Services Apple gave access to CloudKit to non Apple devices / applications / services. Access to public CloudKit db as admin user ( read / write permissions ) is available with generated token and does not require any user login.

CloudKit could be used as passive backend that iOS apps interact with. Maintained by backend JVM service that process requests and updates data in streaming or batch manner.

Interesting to me was to develop simple exchange which will give iOS application possibility to write data which needs to be processed by backend service offline and as result of processing other CloudKit “table” should be updated.

First thing was to declare new RecordType, here called Exchange with following fields

type: String
timestamp: Date/Time
marked: Int(64)
data: Asset

type is intended for separation of different kinds of exchange messages, timestamp represents time when message was put in queue, marked should be used to mark processed messages, which later can be removed, data is Asset which will contain actual message. Message could be anything, in my case JSON.

Initialization of CloudKit exchange from iOS application

let container = CKContainer(identifier: containerName)
self.database = container.publicCloudDatabase
self.exchangeRecordType = exchangeRecordType // "Exchange"

Queue message to exchange, maybe queue is to strong word here

func put(
    _ type: String,
    dataUrl: URL,
    _ handler: @escaping (_ status: CloudKitExchangeRequestStatus) -> Void) -> Void {
    let exchangeMessage = CKRecord(recordType: self.exchangeRecordType)
    exchangeMessage.setObject(Date() as CKRecordValue?, forKey: "timestamp")
    exchangeMessage.setObject(0 as CKRecordValue?, forKey: "marked")
    exchangeMessage.setObject(type as CKRecordValue?, forKey: "type")
    let dataAsset = CKAsset(fileURL: dataUrl)
    exchangeMessage.setObject(dataAsset, forKey: "data") { (record, error) in
        if record != nil && error == nil {
            DispatchQueue.main.async {
        } else {
            DispatchQueue.main.async {

With CloudKitExchange in place posting of message is easy

cloudKitExchange.put("locations", dataUrl: FileUtils.getFullPathUrlInDocuments("locations.json")) { (status) in
    if status == .ok {
        self.statusLabel.text = "done"
    } else {
        self.statusLabel.text = "fail"

Full code for CloudKitExchange can be found on github

To retrieve message from JVM backend app I was using simple CloudKit client, developed for Clojure ( clj-cloudkit  )

(defn pop [client type]
  (let [records (cloudkit/records-query
                    (cloudkit-filter/equals :marked 0)
                    (cloudkit-filter/equals :type type))
                    (cloudkit-sort/descending :timestamp)))]
    (if-let [record (first records)]
      (let [data-url (:downloadURL (:data record))]
        (if-let [data (cloudkit/assets-download
                  (assoc (select-keys record [:marked]) :marked 1)

Full code for clj-client end exchange can be found under github

boot-clj for building simple script

# install

sudo bash -c "cd /usr/local/bin && curl -fsSLo boot && chmod 755 boot"

# simple script ( )

#!/usr/bin/env boot
(set-env! :dependencies '[[org.apache.commons/commons-lang3 "3.5"]])
(println (org.apache.commons.lang3.StringUtils/join ["a" "b" "c"] ","))

# run


Apple CloudKit Server to Server request in JVM environment


, , , , , , , ,

Three things are needed for this code to work, private key ( as hex ), keyid ( one given in CloudKit Dashboard and container id. Code demonstrates call to /users/current route which returns assigned userid.

To get private key in required format from one created following Apple documentation use following command:

 openssl ec -in key.pem -noout -text

Concatenate private key by removing colons.

Container id should start with iCloud.

Path for me to make this work was, first ensure that node.js code is working, play with JVM setup, I used super useful ECDSA Online checker to first ensure signing is working…



Clojure code:

(require '[clj-http.client :as http])
(require '[ :as json])

(defn create-signature-fn [private-key-hex]
  (let [keypair-generator ( "EC")
        prime256v1-gen-param-spec (new "secp256r1")]
    (.initialize keypair-generator prime256v1-gen-param-spec)
    (let [ec-params (.getParams (.getPrivate (.generateKeyPair keypair-generator)))
          private-key-bigint (new java.math.BigInteger private-key-hex 16)
          private-key-specs (new private-key-bigint ec-params)
          key-factory ( "EC")
          private-key (.generatePrivate key-factory private-key-specs)
          signature ( "SHA256withECDSA")
          base64-encoder (java.util.Base64/getEncoder)]
      (.initSign signature private-key)
      (fn [body]
        (.update signature (.getBytes body))
        (.encodeToString base64-encoder (.sign signature))))))

(defn user-request [private-key-hex key-id cloudkit-container]
  (let [simple-date-formatter (new java.text.SimpleDateFormat "yyyy-MM-dd'T'HH:mm:ss'Z'")
        timezone (java.util.TimeZone/getTimeZone "UTC")
        base64-encoder (java.util.Base64/getEncoder)
        sha256-digest ( "SHA-256")
        signature-fn (create-signature-fn private-key-hex)]
    (.setTimeZone simple-date-formatter timezone)
    (let [request-date-iso (.format simple-date-formatter (new java.util.Date))
          subpath (str "/database/1/" cloudkit-container "/development/public/users/current")
          body ""
          body-sha256 (.digest sha256-digest (.getBytes body))
          body-sha256-base64 (.encodeToString base64-encoder body-sha256)
          sign-request (str request-date-iso ":" body-sha256-base64 ":" subpath)
          signature (signature-fn sign-request)
          headers {
                    "X-Apple-CloudKit-Request-KeyID" key-id
                    "X-Apple-CloudKit-Request-ISO8601Date" request-date-iso
                    "X-Apple-CloudKit-Request-SignatureV1" signature}]
      (let [response (http/get
                       (str "" subpath)
                         :headers headers
                         :insecure? true})]
        (json/read-str (:body response))))))

Java code:

import com.fasterxml.jackson.databind.ObjectMapper;

import java.math.BigInteger;
import java.text.SimpleDateFormat;
import java.util.*;

public class Sample {
    public static void main(String[] args) throws Exception {
        String privateKeyHex = args[0];
        String keyId = args[1];
        String cloudKitContainer = args[2];

        KeyPairGenerator keyPairGenerator = KeyPairGenerator.getInstance("EC");
        AlgorithmParameterSpec prime256v1ParamSpec = new ECGenParameterSpec("secp256r1");


        ECParameterSpec parameterSpec = ((ECKey)keyPairGenerator.generateKeyPair().getPrivate()).getParams();

        BigInteger privateKeyInt = new BigInteger(privateKeyHex, 16);

        ECPrivateKeySpec privateKeySpec = new ECPrivateKeySpec(privateKeyInt, parameterSpec);
        KeyFactory keyFactory = KeyFactory.getInstance("EC");
        PrivateKey privateKey = keyFactory.generatePrivate(privateKeySpec);
        Signature signature = Signature.getInstance("SHA256withECDSA");


        SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss'Z'");

        Base64.Encoder encoder = Base64.getEncoder();

        MessageDigest sha256Digest = MessageDigest.getInstance("SHA-256");

        String requestDateIso = simpleDateFormat.format(new Date());
        String subPath = "/database/1/" + cloudKitContainer +  "/development/public/users/current";
        String body = "";
        byte[] bodySha256 = sha256Digest.digest(body.getBytes());
        String bodySha256Base64 = encoder.encodeToString(bodySha256);

        String signRequest = requestDateIso + ":" + bodySha256Base64 + ":" + subPath;
        String signedRequest = encoder.encodeToString(signature.sign());

        URL url = new URL("" + subPath);

        HttpURLConnection connection = (HttpURLConnection)url.openConnection();


        connection.setRequestProperty("X-Apple-CloudKit-Request-KeyID", keyId);
        connection.setRequestProperty("X-Apple-CloudKit-Request-ISO8601Date", requestDateIso);
        connection.setRequestProperty("X-Apple-CloudKit-Request-SignatureV1", signedRequest);

        ObjectMapper objectMapper = new ObjectMapper();
        Map<String, Object> response = objectMapper.readValue(connection.getInputStream(),



time should be UTC formatted with fixed Z instead of zone offset

Working node.js code for reference:

(function() {

	var fs = require('fs');
	var crypto = require('crypto');

	var message = "test";

	var key = fs.readFileSync("key.pem", "utf8");

	var signature = crypto.createSign("sha256");

	var signedMessage = signature.sign(key, "base64");




[apple] Composing Web Service Requests

[stackoverflow] CloudKit Server-to-Server authentication

ECDSA Online checker

ANDREA CORBELLINI – Elliptic Curve Cryptography: a gentle introduction

[nodejs] Clout Kit Web Services using server to server token

Share common code between Clojure projects


, , , , ,

If you want to create some common Clojure code that will be reused between multiple projects but code is not yet mature to become independent library lein has solution for that. It’s called checkouts. There is tons of documentation and other posts online.Something I didn’t get at first is how dependencies of shared ( common ) project are resolved.

Let’s assume following organization:

clojure-repl - common code project / one that will be shared
photo-db-server - project you are working on

First you should go into clojure-repl and do lein install, which will generate pom file and install locally jar and pom to maven repo. In case you want Java like naming you should use com.domain/project-name inside project.clj for project name.
After that go to photo-db-server and add to dependencies:

[com.domain/project-name "version"]

Next comes checkouts, create checkouts directory in project root ( in photo-db-server ). Link root of common project inside checkouts directory, photo-db-server directory structure should look like:

# project.clj
# checkouts
## clojure-repl ( link to ../../clojure-repl )

By adding dependency to photo-db-server project to clojure-repl we ensured all dependencies of clojure-repl will be followed ( by examining pom in local maven repo )
You can examine classpath with:

lein classpath

It should contain both dependencies from common project and project you are working on.