runtime dependency resolution using Maven

Tags

, , , , , , ,

Use Maven to gather dependencies of given artifact. Download missing dependencies into local repository. Build class path. Using URLClassLoader prepare environment for running of artifact’s main class in host JVM.  Host JVM will have only classes needed for interaction with Maven initially loaded.

Example code on github repo

Preparation of simple Jetty example:

cd apps/simple-http-server/
mvn clean install

Running of host JVM and dynamic Jetty example loading:

cd playground/maven-testing/
mvn clean package
java -cp target/maven-testing-1.0-SNAPSHOT.jar com.mungolab.playground.maven.JettyServerExample

Testing:

http://localhost:7071/test-route

 
Code of JettyServerExample:

public class JettyServerExample {
    public static void main(String[] args) {

        System.out.println("Running Jetty example");

        MavenUtils.runArtifactMain(
                "com.mungolab.saturday-project",
                "simple-http-server",
                "1.0-SNAPSHOT",
        	"com.mungolab.sp.apps.httpserver",
        	"ServerMain");
    }
}

Dependency tree for maven-testing project:

 com.mungolab.playground:maven-testing:jar:1.0-SNAPSHOT
 +- org.eclipse.aether:aether-impl:jar:1.1.0:compile
 |  +- org.eclipse.aether:aether-api:jar:1.1.0:compile
 |  +- org.eclipse.aether:aether-spi:jar:1.1.0:compile
 |  \- org.eclipse.aether:aether-util:jar:1.1.0:compile
 +- org.eclipse.aether:aether-connector-basic:jar:1.1.0:compile
 +- org.eclipse.aether:aether-transport-file:jar:1.1.0:compile
 +- org.eclipse.aether:aether-transport-http:jar:1.1.0:compile
 |  +- org.apache.httpcomponents:httpclient:jar:4.3.5:compile
 |  |  +- org.apache.httpcomponents:httpcore:jar:4.3.2:compile
 |  |  \- commons-codec:commons-codec:jar:1.6:compile
 |  \- org.slf4j:jcl-over-slf4j:jar:1.6.2:compile
 |     \- org.slf4j:slf4j-api:jar:1.6.2:compile
 \- org.apache.maven:maven-aether-provider:jar:3.3.9:compile
    +- org.apache.maven:maven-model:jar:3.3.9:compile
    +- org.apache.maven:maven-model-builder:jar:3.3.9:compile
    |  +- org.codehaus.plexus:plexus-interpolation:jar:1.21:compile
    |  +- org.apache.maven:maven-artifact:jar:3.3.9:compile
    |  +- org.apache.maven:maven-builder-support:jar:3.3.9:compile
    |  \- com.google.guava:guava:jar:18.0:compile
    +- org.apache.maven:maven-repository-metadata:jar:3.3.9:compile
    +- org.codehaus.plexus:plexus-component-annotations:jar:1.6:compile
    +- org.codehaus.plexus:plexus-utils:jar:3.0.22:compile
    \- org.apache.commons:commons-lang3:jar:3.4:compile

Dependency tree for Jetty example project:

 com.mungolab.saturday-project:simple-http-server:jar:1.0-SNAPSHOT
 \- org.eclipse.jetty:jetty-server:jar:9.2.24.v20180105:compile
    +- javax.servlet:javax.servlet-api:jar:3.1.0:compile
    +- org.eclipse.jetty:jetty-http:jar:9.2.24.v20180105:compile
    |  \- org.eclipse.jetty:jetty-util:jar:9.2.24.v20180105:compile
    \- org.eclipse.jetty:jetty-io:jar:9.2.24.v20180105:compile

Given example relies on Aether project for coordination with  Maven.

Links:
http://maven.apache.org/plugins/maven-dependency-plugin/get-mojo.html
https://stackoverflow.com/questions/39638138/find-all-direct-dependencies-of-an-artifact-on-maven-central
https://stackoverflow.com/questions/40813062/maven-get-all-dependencies-programmatically

Advertisements

running simple Java HTTP server on OSv and EC2

Tags

, , , , , ,

Be able to run Java App on top of EC2 with OSv ( as minimalistic operating system as possible ) . Create AMI with custom jar and start application once machine is up.

Using Capstan should be way to go, create new image, test it on QEMU or Virtual Box and once ready produce EC2 image. That path didn’t go well for me, I had issues installing Capstan, running sample image initially on Mac and later trying on EC2.

Instead I went in different direction, used one of already available OSv AMIs. I used OSv-Tomcat to ensure I have Java ready. OSv comes with Java 1.7. EC2 instance used was m3.medium. I was unable to get dashboard running on m4.medium. Easiest way to ensure instance is up and ready is to check Dashboard running on 8000. Make sure port 8000 is allowed in Security Group ( for my HTTP server I also used additional 7071 port ). Tomcat which is started out of box is running on 8081.

Next step is customization. For all required steps Dashboard REST API was enough. Create uber jar of custom http server, I was using simple-http-server given in links, running mvn clean package. Upload uber jar over Dashboard REST File API.

Use Dashboard REST OS API to change startup command, original command was

java.so io.osv.MultiJarLoader -mains /etc/javamains

to ( http server uber jar is named app.jar on OSv FS )

java.so -jar app.jar

Use Dashboard REST OS API to restart OS. Once OS is restarted custom http server should be available on 7071 port.

http://:7071/test

should return

Hello World on: /test.

 

Create AMI from EC2 EBS volume and use it to spin tons of simple http servers 🙂

Links:

Porting-Java-applications-to-OSv

capstan-example-java

simple-http-server

OSv

saturday-project

chunked-seq, sequence and mapcat

Tags

, , , ,

(first 
  (map 
    (fn [element] (println "realizing" element) element) 
    (range 0 100)))
> realizing 0
> realizing 1
> realizing 2
...
> realizing 30
> realizing 31
=> 0

versus:

(first 
  (map 
    (fn [element] (println "realizing" element) element) 
    (range)))
> realizing 0
=> 0

versus:

(first 
  (map 
    (fn [element] (println "realizing" element) element) 
    '(0 1 2 3 4)))
> realizing 0
=> 0

chunked-seq:

(chunked-seq? (range 0 4))
=> true
(chunked-seq? '(0 1 2 3))
=> false
(chunked-seq? (range))
=> false

mapcat, transducers, sequence:

(take
  2
  (sequence
    (mapcat (fn [element] (println "realizing" element) [element]))
    (range)))
> realizing 0
> realizing 1
> realizing 2
...
> realizing 31
> realizing 32
=> (0 1)

versus:

(sequence 
  (comp 
    (mapcat (fn [element] (println "realizing" element) [element])) 
    (take 2)) 
  (range))
> realizing 0
> realizing 1
=> (0 1)

versus:

(take
  2
  (mapcat
    (fn [element] (println "realizing" element) [element])
    (range)))
> realizing 0
> realizing 1
> realizing 2
> realizing 3
=> (0 1)

array -> dictionary with key fn

Tags

, , ,

class Utils {
    class func indexSequence<K, V>(
        _ sequence: [V],
        _ keyComputeFn: (_ value: V) -> K) -> Dictionary<K, V> {
        
        return sequence.reduce(Dictionary<K,V>()) { (result, value) in
            var result = result
            result[keyComputeFn(value)] = value
            return result
        }
    }
}

test:

let sequence = ["a", "ab", "abc"]
let wordCountFn: (String) -> Int = { word in word.count }

let result = Utils.indexSequence(sequence, wordCountFn)
// [2: "ab", 3: "abc", 1: "a"]

Clojure type hinting

Tags

, , , , , ,

Clojure code snipet

(let [headers (vector "A" "B" "C")]
  (println "with type hint")
  (.indexOf ^java.util.List headers "B")
  (println "without type hint")
  (.indexOf headers "B"))

generates following byte code

 0: getstatic     #15                 // Field const__0:Lclojure/lang/Var;
 3: invokevirtual #20                 // Method clojure/lang/Var.getRawRoot:()Ljava/lang/Object;
 6: checkcast     #22                 // class clojure/lang/IFn
 9: ldc           #24                 // String A
11: ldc           #26                 // String B
13: ldc           #28                 // String C
15: invokeinterface #32,  4           // InterfaceMethod clojure/lang/IFn.invoke:(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
20: astore_0
21: getstatic     #35                 // Field const__1:Lclojure/lang/Var;
24: invokevirtual #20                 // Method clojure/lang/Var.getRawRoot:()Ljava/lang/Object;
27: checkcast     #22                 // class clojure/lang/IFn
30: ldc           #37                 // String with type hint
32: invokeinterface #40,  2           // InterfaceMethod clojure/lang/IFn.invoke:(Ljava/lang/Object;)Ljava/lang/Object;
37: pop
38: aload_0
39: checkcast     #42                 // class java/util/List
42: ldc           #26                 // String B
44: invokeinterface #46,  2           // InterfaceMethod java/util/List.indexOf:(Ljava/lang/Object;)I
49: invokestatic  #52                 // Method java/lang/Integer.valueOf:(I)Ljava/lang/Integer;
52: pop
53: getstatic     #35                 // Field const__1:Lclojure/lang/Var;
56: invokevirtual #20                 // Method clojure/lang/Var.getRawRoot:()Ljava/lang/Object;
59: checkcast     #22                 // class clojure/lang/IFn
62: ldc           #54                 // String without type hint
64: invokeinterface #40,  2           // InterfaceMethod clojure/lang/IFn.invoke:(Ljava/lang/Object;)Ljava/lang/Object;
69: pop
70: aload_0
71: aconst_null
72: astore_0
73: ldc           #55                 // String indexOf
75: iconst_1
76: anewarray     #57                 // class java/lang/Object
79: dup
80: iconst_0
81: ldc           #26                 // String B
83: aastore
84: invokestatic  #63                 // Method clojure/lang/Reflector.invokeInstanceMethod:(Ljava/lang/Object;Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/Object;
87: areturn

focus should be on

38: aload_0
39: checkcast     #42                 // class java/util/List
42: ldc           #26                 // String B
44: invokeinterface #46,  2           // InterfaceMethod java/util/List.indexOf:(Ljava/lang/Object;)I
49: invokestatic  #52                 // Method java/lang/Integer.valueOf:(I)Ljava/lang/Integer;

versus

70: aload_0
71: aconst_null
72: astore_0
73: ldc           #55                 // String indexOf
75: iconst_1
76: anewarray     #57                 // class java/lang/Object
79: dup
80: iconst_0
81: ldc           #26                 // String B
83: aastore
84: invokestatic  #63                 // Method clojure/lang/Reflector.invokeInstanceMethod:(Ljava/lang/Object;Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/Object;

One of ways to generate given sample is to create new lein project, define test namespace and add given clojure snipet. Add namespace to :aot in project.clj. Perform lein jar, unpack jar, and use javap -private -c -verbose on one of namespace generated class files …

drawing the world map

Tags

, , , , , ,

continents

prepare dataset
download and extract to /tmp/dataset/naturalearthdata.com, you should have ne_110m_geography_regions_polys directory inside naturalearthdata.com containing ne_110m_geography_regions_polys.shp and other files.

checkout code and run repl

git clone git@github.com:vanjakom/clj-geo.git
cd clj-geo
lein repl

 

create continents image

(require 'clj-geo.visualization.raster)
(require 'clj-geo.env)
(require 'clj-common.2d)
(require 'clj-common.localfs)
(binding [clj-geo.env/*dataset-path* ["tmp" "dataset"]]
  (let [continent-image (clj-geo.visualization.raster/create-continent-image)]
    (clj-common.2d/write-to-stream
      continent-image
      (clj-common.localfs/output-stream ["tmp" "continents.bmp"]))))

multiple-reduce, clojure macro

Tags

, , , , ,

(defmacro multiple-reduce
  [bindings coll]
  `(reduce
     (fn [state# value#]
       (apply
         array-map
         (apply
           concat
           (map
             (fn [[key# default# func#]]
               [key# (func# (get state# key# default#) value#)])
             (partition
               3
               ~bindings)))))
     {}
     ~coll))

Macro for doing multiple parallel aggregations on objects sequence resulting in single aggregate object.

multiple-reduce is defined under clj-common project

Example:

(multiple-reduce
  [
    :inc 0 #(+ %1 (inc %2))
    :dec 0 #(+ %1 (dec %2))]
  '(1 2 3))

will result with

{:inc 9, :dec 3}

Each binding is defined with  keyword that will contain aggregated value after aggregation is finished, initial value and aggregate function ( f(state, value) -> state ). Aggregate function is function of two args, aggregated value ( state ) and increment value ( value ) which with each iteration returns new state.

Example usage from my project

(let [first-location (first locations-seq)
      aggregate (multiple-reduce
                  [
                    :max-longitude (:longitude first-location) (partial max-aggregate :longitude)
                    :min-longitude (:longitude first-location) (partial min-aggregate :latitude)
                    :max-latitude (:latitude first-location) (partial max-aggregate :latitude)
                    :min-latitude (:latitude first-location) (partial min-aggregate :latitude)
                    :distance [0 first-location] distance-aggregate
                    :last-location first-location last-location-aggregate
                    :activity-duration [0 first-location] duration-without-pause]
                    locations-seq)]
  
  ;continue computation on aggregate

  )

Where distance-aggregate is defined as

(defn distance-aggregate [[distance previous-location] location]
  [(+ distance (geo/distance-two-locations previous-location location)) location])

CloudKit as exchange between iOS app and JVM backend service

cloudkit-exchange

With CloudKit Web Services Apple gave access to CloudKit to non Apple devices / applications / services. Access to public CloudKit db as admin user ( read / write permissions ) is available with generated token and does not require any user login.

CloudKit could be used as passive backend that iOS apps interact with. Maintained by backend JVM service that process requests and updates data in streaming or batch manner.

Interesting to me was to develop simple exchange which will give iOS application possibility to write data which needs to be processed by backend service offline and as result of processing other CloudKit “table” should be updated.

First thing was to declare new RecordType, here called Exchange with following fields

type: String
timestamp: Date/Time
marked: Int(64)
data: Asset

type is intended for separation of different kinds of exchange messages, timestamp represents time when message was put in queue, marked should be used to mark processed messages, which later can be removed, data is Asset which will contain actual message. Message could be anything, in my case JSON.

Initialization of CloudKit exchange from iOS application

let container = CKContainer(identifier: containerName)
self.database = container.publicCloudDatabase
self.exchangeRecordType = exchangeRecordType // "Exchange"

Queue message to exchange, maybe queue is to strong word here

func put(
    _ type: String,
    dataUrl: URL,
    _ handler: @escaping (_ status: CloudKitExchangeRequestStatus) -> Void) -> Void {
    
    let exchangeMessage = CKRecord(recordType: self.exchangeRecordType)
    exchangeMessage.setObject(Date() as CKRecordValue?, forKey: "timestamp")
    exchangeMessage.setObject(0 as CKRecordValue?, forKey: "marked")
    exchangeMessage.setObject(type as CKRecordValue?, forKey: "type")
    
    let dataAsset = CKAsset(fileURL: dataUrl)
    exchangeMessage.setObject(dataAsset, forKey: "data")
    
    self.database.save(exchangeMessage) { (record, error) in
        if record != nil && error == nil {
            DispatchQueue.main.async {
                handler(.fail)
            }
        } else {
            DispatchQueue.main.async {
                handler(.fail)
            }
        }
    }
}

With CloudKitExchange in place posting of message is easy

cloudKitExchange.put("locations", dataUrl: FileUtils.getFullPathUrlInDocuments("locations.json")) { (status) in
    if status == .ok {
        self.statusLabel.text = "done"
    } else {
        self.statusLabel.text = "fail"
    }
}

Full code for CloudKitExchange can be found on github

To retrieve message from JVM backend app I was using simple CloudKit client, developed for Clojure ( clj-cloudkit  )

(defn pop [client type]
  (let [records (cloudkit/records-query
                  client
                  *exchange-record-type*
                  (list
                    (cloudkit-filter/equals :marked 0)
                    (cloudkit-filter/equals :type type))
                  (list
                    (cloudkit-sort/descending :timestamp)))]
    (if-let [record (first records)]
      (let [data-url (:downloadURL (:data record))]
        (if-let [data (cloudkit/assets-download
                        client
                        data-url)]
          (do
            (cloudkit/records-modify
              client
              (list
                (cloudkit-operation/update
                  (assoc (select-keys record [:marked]) :marked 1)
                  *exchange-record-type*)))
            data))))))

Full code for clj-client end exchange can be found under github

boot-clj for building simple script

# install

sudo bash -c "cd /usr/local/bin && curl -fsSLo boot https://github.com/boot-clj/boot-bin/releases/download/latest/boot.sh && chmod 755 boot"

# simple script ( simple.sh )

#!/usr/bin/env boot
(set-env! :dependencies '[[org.apache.commons/commons-lang3 "3.5"]])
(println (org.apache.commons.lang3.StringUtils/join ["a" "b" "c"] ","))

# run

./simple.sh

Apple CloudKit Server to Server request in JVM environment

Tags

, , , , , , , ,

Three things are needed for this code to work, private key ( as hex ), keyid ( one given in CloudKit Dashboard and container id. Code demonstrates call to /users/current route which returns assigned userid.

To get private key in required format from one created following Apple documentation use following command:

 openssl ec -in key.pem -noout -text

Concatenate private key by removing colons.

Container id should start with iCloud.

Path for me to make this work was, first ensure that node.js code is working, play with JVM setup, I used super useful ECDSA Online checker to first ensure signing is working…

(user-request
	"00b51d4..."
	"b6367f3..."
	"iCloud....")

 

Clojure code:

(require '[clj-http.client :as http])
(require '[clojure.data.json :as json])

(defn create-signature-fn [private-key-hex]
  (let [keypair-generator (java.security.KeyPairGenerator/getInstance "EC")
        prime256v1-gen-param-spec (new java.security.spec.ECGenParameterSpec "secp256r1")]
    (.initialize keypair-generator prime256v1-gen-param-spec)
    (let [ec-params (.getParams (.getPrivate (.generateKeyPair keypair-generator)))
          private-key-bigint (new java.math.BigInteger private-key-hex 16)
          private-key-specs (new java.security.spec.ECPrivateKeySpec private-key-bigint ec-params)
          key-factory (java.security.KeyFactory/getInstance "EC")
          private-key (.generatePrivate key-factory private-key-specs)
          signature (java.security.Signature/getInstance "SHA256withECDSA")
          base64-encoder (java.util.Base64/getEncoder)]
      (.initSign signature private-key)
      (fn [body]
        (.update signature (.getBytes body))
        (.encodeToString base64-encoder (.sign signature))))))

(defn user-request [private-key-hex key-id cloudkit-container]
  (let [simple-date-formatter (new java.text.SimpleDateFormat "yyyy-MM-dd'T'HH:mm:ss'Z'")
        timezone (java.util.TimeZone/getTimeZone "UTC")
        base64-encoder (java.util.Base64/getEncoder)
        sha256-digest (java.security.MessageDigest/getInstance "SHA-256")
        signature-fn (create-signature-fn private-key-hex)]
    (.setTimeZone simple-date-formatter timezone)
    (let [request-date-iso (.format simple-date-formatter (new java.util.Date))
          subpath (str "/database/1/" cloudkit-container "/development/public/users/current")
          body ""
          body-sha256 (.digest sha256-digest (.getBytes body))
          body-sha256-base64 (.encodeToString base64-encoder body-sha256)
          sign-request (str request-date-iso ":" body-sha256-base64 ":" subpath)
          signature (signature-fn sign-request)
          headers {
                    "X-Apple-CloudKit-Request-KeyID" key-id
                    "X-Apple-CloudKit-Request-ISO8601Date" request-date-iso
                    "X-Apple-CloudKit-Request-SignatureV1" signature}]
      (let [response (http/get
                       (str "https://api.apple-cloudkit.com" subpath)
                       {
                         :headers headers
                         :insecure? true})]
        (json/read-str (:body response))))))

Java code:

import com.fasterxml.jackson.databind.ObjectMapper;

import java.math.BigInteger;
import java.net.HttpURLConnection;
import java.net.URL;
import java.security.*;
import java.security.interfaces.ECKey;
import java.security.spec.AlgorithmParameterSpec;
import java.security.spec.ECGenParameterSpec;
import java.security.spec.ECParameterSpec;
import java.security.spec.ECPrivateKeySpec;
import java.text.SimpleDateFormat;
import java.util.*;

public class Sample {
    public static void main(String[] args) throws Exception {
        String privateKeyHex = args[0];
        String keyId = args[1];
        String cloudKitContainer = args[2];

        KeyPairGenerator keyPairGenerator = KeyPairGenerator.getInstance("EC");
        AlgorithmParameterSpec prime256v1ParamSpec = new ECGenParameterSpec("secp256r1");

        keyPairGenerator.initialize(prime256v1ParamSpec);

        ECParameterSpec parameterSpec = ((ECKey)keyPairGenerator.generateKeyPair().getPrivate()).getParams();

        BigInteger privateKeyInt = new BigInteger(privateKeyHex, 16);

        ECPrivateKeySpec privateKeySpec = new ECPrivateKeySpec(privateKeyInt, parameterSpec);
        KeyFactory keyFactory = KeyFactory.getInstance("EC");
        PrivateKey privateKey = keyFactory.generatePrivate(privateKeySpec);
        Signature signature = Signature.getInstance("SHA256withECDSA");

        signature.initSign(privateKey);

        SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss'Z'");
        simpleDateFormat.setTimeZone(TimeZone.getTimeZone("UTC"));

        Base64.Encoder encoder = Base64.getEncoder();

        MessageDigest sha256Digest = MessageDigest.getInstance("SHA-256");

        String requestDateIso = simpleDateFormat.format(new Date());
        String subPath = "/database/1/" + cloudKitContainer +  "/development/public/users/current";
        String body = "";
        byte[] bodySha256 = sha256Digest.digest(body.getBytes());
        String bodySha256Base64 = encoder.encodeToString(bodySha256);

        String signRequest = requestDateIso + ":" + bodySha256Base64 + ":" + subPath;
        signature.update(signRequest.getBytes());
        String signedRequest = encoder.encodeToString(signature.sign());

        URL url = new URL("https://api.apple-cloudkit.com" + subPath);

        HttpURLConnection connection = (HttpURLConnection)url.openConnection();

        connection.setRequestMethod("GET");

        connection.setRequestProperty("X-Apple-CloudKit-Request-KeyID", keyId);
        connection.setRequestProperty("X-Apple-CloudKit-Request-ISO8601Date", requestDateIso);
        connection.setRequestProperty("X-Apple-CloudKit-Request-SignatureV1", signedRequest);

        ObjectMapper objectMapper = new ObjectMapper();
        Map<String, Object> response = objectMapper.readValue(connection.getInputStream(),
                objectMapper.getTypeFactory().constructMapType(
                    HashMap.class,
                    String.class,
                    Object.class));

        System.out.println(response.get("userRecordName"));
    }
}

Notes:

time should be UTC formatted with fixed Z instead of zone offset

Working node.js code for reference:

(function() {
	console.log("starting");

	var fs = require('fs');
	var crypto = require('crypto');

	var message = "test";

	var key = fs.readFileSync("key.pem", "utf8");

	var signature = crypto.createSign("sha256");
	signature.update(message);

	var signedMessage = signature.sign(key, "base64");

	console.log(signedMessage);
})();

 

Links:

[apple] Composing Web Service Requests

[stackoverflow] CloudKit Server-to-Server authentication

ECDSA Online checker

ANDREA CORBELLINI – Elliptic Curve Cryptography: a gentle introduction

[nodejs] Clout Kit Web Services using server to server token