Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

run parameters prototype #1121

Open
wants to merge 61 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
38fb15a
wip
0xbase12 Jul 12, 2017
5c349ea
wip
0xbase12 Jul 12, 2017
90be0aa
wip
0xbase12 Jul 12, 2017
0b71e24
Merge branch 'master' into issue#1112-run-paramtetes-prototype
0xbase12 Jul 12, 2017
30e8445
working sse prototype
0xbase12 Jul 13, 2017
ae09522
sse and zookeeper working
0xbase12 Jul 14, 2017
f42e004
Merge branch 'master' into issue#1112-run-paramtetes-prototype
0xbase12 Jul 18, 2017
c816f8a
run with action prototype working and some unit test
0xbase12 Jul 18, 2017
6cb9db0
wip
0xbase12 Jul 19, 2017
8f0ed09
wip run parameters working with zookeeper for creation
0xbase12 Jul 20, 2017
28ae980
wip run parameter retrieve value from zookeeper
0xbase12 Jul 20, 2017
5a447b0
wip issue with sse
0xbase12 Jul 21, 2017
56d2acb
fix issue with nginx buffering and accept header logic implemented
0xbase12 Jul 22, 2017
e6c5280
run param edit set value in zk and fix issue with get-version from zo…
0xbase12 Jul 23, 2017
622f147
implement query for run param with sse and json
0xbase12 Jul 24, 2017
cbf8d6d
add unit tests and dependencies
0xbase12 Jul 24, 2017
aa6e37e
simple lifecycle test for run with embded zookeeper server working
0xbase12 Jul 25, 2017
63a813d
Merge branch 'issue#1112-run-paramtetes-prototype' into issue#1111-ne…
0xbase12 Jul 25, 2017
cbfbc7e
minor enhancement and more unit test for start run action
0xbase12 Jul 25, 2017
69d7f19
Merge branch 'master' into issue#1111-new-run-ssclj-resource
0xbase12 Jul 27, 2017
22a4ed8
run parameter life cycle tests and run id should be passed at creatio…
0xbase12 Jul 27, 2017
d83a4bf
minor, vmstate should be set by the client
0xbase12 Jul 27, 2017
5866979
add acl for un parameters
0xbase12 Jul 28, 2017
199d006
rename run-id to run-href
0xbase12 Jul 28, 2017
dbc3d37
SSE data is now the full json of run parameter and little more other …
0xbase12 Jul 28, 2017
585052c
Merge branch 'master' into issue#1111-new-run-ssclj-resource
konstan Aug 28, 2017
39a5dee
names of es fixtures changed
konstan Aug 28, 2017
d3ac0d4
Merge branch 'master' into issue#1111-new-run-ssclj-resource
konstan Aug 28, 2017
65e23f6
Merge branch 'master' into issue#1111-new-run-ssclj-resource
konstan Aug 29, 2017
50916db
add values to run parameter query
0xbase12 Sep 1, 2017
023cc36
add type to run parameter spec
0xbase12 Sep 1, 2017
52075aa
store values in elasticsearch
0xbase12 Sep 1, 2017
65baf4c
Merge branch 'master' into issue#1111-new-run-ssclj-resource
0xbase12 Sep 1, 2017
c12bd33
Merge branch 'master' into issue#1111-new-run-ssclj-resource
0xbase12 Sep 4, 2017
39b7d6b
Merge branch 'issue#1111-new-run-ssclj-resource' of ssh://github.com/…
0xbase12 Sep 4, 2017
266984b
renaming run resource to deployment
0xbase12 Sep 5, 2017
a63d623
update deployment attribute state when run parameter is updated
0xbase12 Sep 5, 2017
591feab
in lifecycle test clean all zk nodes after each test
0xbase12 Sep 6, 2017
afb1b85
FIXME: temporarily adding explicit dependency on clojure/tools.reader
konstan Sep 6, 2017
54a7d0d
Merge branch 'issue#1111-new-run-ssclj-resource' of github.com:slipst…
konstan Sep 6, 2017
d953909
removed unused requires
konstan Sep 6, 2017
cf9faf6
WIP update of state-complete for node instance have special behavior …
0xbase12 Sep 6, 2017
bbf5162
WIP for lock and deployment state move
0xbase12 Sep 8, 2017
f0c8ffd
next-state working and idempotent
0xbase12 Sep 11, 2017
5d509e5
SSE should answer in json
0xbase12 Sep 11, 2017
f7d363f
enhance http accept header interpretation
0xbase12 Sep 11, 2017
2984d4f
enhance code and fix deployment-href in deployment parameter
0xbase12 Sep 11, 2017
24dfaf1
rename deployment-href to deployment in deployment parameter and minor
0xbase12 Sep 11, 2017
870325b
Merge branch 'master' into issue#1111-new-run-ssclj-resource
0xbase12 Sep 12, 2017
86a1d2a
abort deployment possible and creation of deployment from java code
0xbase12 Sep 14, 2017
bead98d
fix deployment spec, add abort
0xbase12 Sep 14, 2017
54d863c
new header auth for java, ssclj deployment template
0xbase12 Sep 15, 2017
d923418
fix authentification regression
0xbase12 Sep 18, 2017
ce5973e
Merge branch 'master' into issue#1111-new-run-ssclj-resource
konstan Sep 18, 2017
0578af4
add conversion from java run to deployment and fake http server for t…
0xbase12 Sep 18, 2017
af9e526
Merge branch 'master' into issue#1111-new-run-ssclj-resource
0xbase12 Sep 18, 2017
d3d881a
Merge branch 'issue#1111-new-run-ssclj-resource' of ssh://github.com/…
0xbase12 Sep 18, 2017
41a1327
fix lifecycle tests for deployment
0xbase12 Sep 19, 2017
a683faf
set java runtime parameter are propagated to ssclj and dyn creation o…
0xbase12 Sep 19, 2017
daaa39f
terminate action on deployment to be continued
0xbase12 Sep 20, 2017
95a3763
minor
0xbase12 Sep 21, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
run parameter life cycle tests and run id should be passed at creatio…
…n time
  • Loading branch information
0xbase12 committed Jul 27, 2017

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit 22a4ed893f5b526f93236f24b616b83616c29eb1
6 changes: 2 additions & 4 deletions ssclj/jar/src/com/sixsq/slipstream/ssclj/resources/run.clj
Original file line number Diff line number Diff line change
@@ -57,9 +57,7 @@

(defmethod crud/new-identifier resource-name
[json _]
#_json ;TODO uncomment because the run-id should be consistent with SlipStream java server
(assoc json :id (str resource-url "/" (u/random-uuid))) ;TODO REMOVE this line
)
json)

(defn create-parameter [identity run-parameter]
(try
@@ -89,7 +87,7 @@
(defn add-impl [{body :body :as request}]
(a/can-modify? {:acl collection-acl} request)
(let [new-run (-> body
u/strip-service-attrs
(dissoc :created :updated :resourceURI :operations)
(crud/new-identifier resource-name)
(assoc :resourceURI resource-uri)
u/update-timestamps
Original file line number Diff line number Diff line change
@@ -11,6 +11,7 @@
[com.sixsq.slipstream.util.response :as r]
[com.sixsq.slipstream.db.impl :as db]
[clojure.tools.logging :as log]
[com.sixsq.slipstream.ssclj.util.log :as logu]
[com.sixsq.slipstream.ssclj.util.sse :as sse]
[clojure.core.async :as async]
[zookeeper :as zk]
@@ -73,20 +74,28 @@
node-path (zkru/parameter-znode-path run-id node-name node-index name)]
(uzk/create-all node-path :persistent? true)
(uzk/set-data node-path value)
response
))
response))

(defmethod crud/add resource-name
[request]
(add-impl request))

(defn watch-fn [event-ch id name {:keys [event-type path :as zk-event]}]
(defn transiant-watch-fn [event-ch id name {:keys [event-type path :as zk-event]}]
(when (= event-type :NodeDataChanged)
(let [event {:id id
:name name
:data (uzk/get-data path :watcher (partial watch-fn event-ch id name))}]
(async/>!! event-ch event))))
(let [data (uzk/get-data path)]
(sse/send-event id name data event-ch)
(async/close! event-ch))))

(defn persistent-watch-fn [event-ch id name {:keys [event-type path :as zk-event]}]
(when (= event-type :NodeDataChanged)
(let [data (uzk/get-data path :watcher (partial persistent-watch-fn event-ch id name))]
(sse/send-event id name data event-ch))))

(defn send-event-and-set-watcher
[event-ch watch-fn {id :id name :name :as run-parameter}]
(let [node-path (zkru/run-parameter-znode-path run-parameter)
value (uzk/get-data node-path :watcher (partial watch-fn event-ch id name))]
(sse/send-event id name value event-ch)))

(defn retrieve-run-parameter [{{uuid :uuid} :params :as request}]
(try
@@ -99,22 +108,13 @@
(catch ExceptionInfo ei
(ex-data ei))))

(defn send-param-value-and-set-watcher [event-ch {id :id name :name :as run-parameter}]
(try ; if znode not found do nothing for the moment
(let [node-path (zkru/run-parameter-znode-path run-parameter)
event {:id id
:name name
:data (uzk/get-data node-path :watcher (partial watch-fn event-ch id name))}]
(async/>!! event-ch event))
(catch Exception e
(log/warn (ex-data e))
)))

(def retrieve-sse-impl
(sse/event-channel-handler
(fn [request response raise event-ch]
(let [{id :id name :name :as run-parameter} (retrieve-run-parameter request)]
(send-param-value-and-set-watcher event-ch run-parameter)))
(let [{id :id name :name :as run-parameter} (retrieve-run-parameter request)
node-path (zkru/run-parameter-znode-path run-parameter)
value (uzk/get-data node-path)]
(send-event-and-set-watcher event-ch transiant-watch-fn run-parameter)))
{:on-client-disconnect #(log/debug "sse/on-client-disconnect: " %)}))

(defn retrieve-json-impl [request]
@@ -128,22 +128,21 @@

(defn edit-impl
[{{uuid :uuid} :params body :body :as request}]
(try
(let [current (-> (str (u/de-camelcase resource-name) "/" uuid)
(db/retrieve request)
(a/can-modify? request))
merged (merge current body)
value (:value merged)]
(-> merged
(u/update-timestamps)
(crud/validate)
(dissoc :value)
(db/edit request))
(when value
(uzk/set-data (zkru/run-parameter-znode-path merged) value)) ;TODO what if znode not found
)
(catch ExceptionInfo ei
(ex-data ei))))
(let [current (-> (str (u/de-camelcase resource-name) "/" uuid)
(db/retrieve request)
(a/can-modify? request))
merged (merge current body)
value (:value merged)]
(-> merged
(u/update-timestamps)
(crud/validate)
(dissoc :value)
(db/edit request))
(when value
(let [name (:name merged)]
(if (contains? #{"state" "vmstate"} name)
(logu/log-and-throw-400 (str "run parameter " name " cannot be set"))
(uzk/set-data (zkru/run-parameter-znode-path merged) value)))))) ;TODO what if znode not found

(defmethod crud/edit resource-name
[request]
@@ -161,7 +160,7 @@
(a/can-view? {:acl collection-acl} request)
(let [options (select-keys request [:identity :query-params :cimi-params :user-name :user-roles])
[count-before-pagination entries] (db/query resource-name options)]
(doall (map (partial send-param-value-and-set-watcher event-ch) entries))))
(doall (map (partial send-event-and-set-watcher event-ch persistent-watch-fn) entries))))
{:on-client-disconnect #(log/debug "sse/on-client-disconnect: " %)}))

(def query-json-impl (std-crud/query-fn resource-name collection-acl collection-uri resource-tag))
Original file line number Diff line number Diff line change
@@ -56,7 +56,7 @@

(s/def :cimi.run/nodes (su/constrained-map keyword? :cimi.run/node))

(def run-attrs {:req-un [#_:cimi.common/id ;TODO uncomment
(def run-attrs {:req-un [:cimi.common/id
:cimi.run/module-resource-uri
:cimi.run/type
:cimi.run/category
Original file line number Diff line number Diff line change
@@ -21,6 +21,7 @@

;; State machine states
(def initial-state "init")

(def valid-transitions
{initial-state ["provisioning"]
"provisioning" ["executing"]
13 changes: 9 additions & 4 deletions ssclj/jar/src/com/sixsq/slipstream/ssclj/util/sse.clj
Original file line number Diff line number Diff line change
@@ -5,8 +5,7 @@
[ring.core.protocols :as ring]
[ring.util.response :as ring-response]
[manifold.stream :as s]
[manifold.deferred :as d])
)
[manifold.deferred :as d]))

(def CRLF "\r\n")
(def EVENT_FIELD "event: ")
@@ -36,7 +35,13 @@
(.append sb CRLF)
(str sb))))

(defn send-event
(defn send-event [id name data event-ch]
(let [event {:id id
:name name
:data data}]
(async/>!! event-ch event)))

(defn- send-event-on-response-ch
[channel name data id put-fn raise]
(try
(put-fn channel (mk-data name data id))
@@ -66,7 +71,7 @@
(if (map? event)
(reduce (fn [agg [k v]] (assoc agg k (str v))) {} event)
{:data (str event)})]
(when (send-event response-channel event-name event-data event-id async/put! raise)
(when (send-event-on-response-ch response-channel event-name event-data event-id async/put! raise)
(recur))))))
(async/close! event-channel)
(async/close! response-channel)
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@
(t/make-ring-app (t/concat-routes routes/final-routes)))

(def valid-entry
{:id (str resource-url "run/dfd34916-6ede-47f7-aaeb-a30ddecbba5b")
{:id (str resource-url "/dfd34916-6ede-47f7-aaeb-a30ddecbba5b")
:resourceURI resource-uri
:module-resource-uri "module/examples/tutorials/service-testing/system/1940"
:category "Deployment"
@@ -69,7 +69,6 @@
(t/body->edn)
(t/is-status 200))]


(is (not (uzk/exists (str ru/znode-separator run-id))))

(let [start-uri (str p/service-context (t/get-op created-run "http://schemas.dmtf.org/cimi/2/action/start"))
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
(ns com.sixsq.slipstream.ssclj.resources.run-parameter-lifecycle-test
(:require
[clojure.test :refer :all]
[clojure.data.json :as json]
[peridot.core :refer :all]
[ring.util.codec :as rc]
[com.sixsq.slipstream.ssclj.resources.run-parameter :refer :all]
[com.sixsq.slipstream.ssclj.resources.lifecycle-test-utils :as t]
[com.sixsq.slipstream.ssclj.middleware.authn-info-header :refer [authn-info-header]]
[com.sixsq.slipstream.ssclj.app.routes :as routes]
[com.sixsq.slipstream.ssclj.app.params :as p]
[com.sixsq.slipstream.ssclj.resources.common.utils :as u]
[com.sixsq.slipstream.ssclj.util.zookeeper :as uzk]
[com.sixsq.slipstream.ssclj.resources.zk.run.utils :as ru]
[com.sixsq.slipstream.ssclj.resources.zk.run.utils :as zkru]))

(use-fixtures :each t/with-test-client-fixture)

(use-fixtures :once t/setup-embedded-zk)

(def base-uri (str p/service-context resource-url))

(defn ring-app []
(t/make-ring-app (t/concat-routes routes/final-routes)))

(def session-user (-> (session (ring-app))
(content-type "application/json")
(header authn-info-header "jane USER ANON")))

(deftest create-run-parameter-xyz
(let [run-id "run/abc34916-6ede-47f7-aaeb-a30ddecbba5b"
valid-entry {:run-id run-id :node-name "machine" :node-index 1 :name "xyz" :value "XYZ"}
znode-path (zkru/run-parameter-znode-path valid-entry)
run-parameter-id (-> session-user
(request base-uri
:request-method :post
:body (json/write-str valid-entry))
(t/body->edn)
(t/is-status 201)
(t/location))
abs-uri (str p/service-context (u/de-camelcase run-parameter-id))
created-run-parameter (-> session-user
(request abs-uri)
(t/body->edn)
(t/is-status 200))]

(is (= "XYZ" (uzk/get-data znode-path)))

(-> session-user
(request abs-uri :request-method :put
:body (json/write-str {:value "newvalue"}))
(t/body->edn)
(t/is-status 200))

(is (= "newvalue" (uzk/get-data znode-path)) "run parameter can be updated")))

(deftest create-run-parameter-state
(let [run-id "run/def34916-6ede-47f7-aaeb-a30ddecbba5b"
valid-entry {:run-id run-id :name "state" :value "init"}
run-parameter-id (-> session-user
(request base-uri
:request-method :post
:body (json/write-str valid-entry))
(t/body->edn)
(t/is-status 201)
(t/location))
abs-uri (str p/service-context (u/de-camelcase run-parameter-id))
created-run-parameter (-> session-user
(request abs-uri)
(t/body->edn)
(t/is-status 200))]

(is (= "init" (uzk/get-data (str ru/znode-separator run-id "/state"))) "znode state is created")

(-> session-user
(request abs-uri :request-method :put
:body (json/write-str {:value "newvalue"}))
(t/body->edn)
(t/is-status 400))

(is (= "init" (uzk/get-data (str ru/znode-separator run-id "/state"))) "run parameter state can't be updated")))