Add test suite and integrate to run on CI

[ckarande]

No description provided.

[binarymist]

What are your thoughts around this @ckarande have non security focused unit tests running and passing or security focused tests (regression maybe?) running and continually failing... as the whole idea of the project is to be broken to start with and then the students fix the vulns?

[ckarande]

@binarymist Sorry for getting back to it after a long time. I didn't mean to ignore it at all.

On thinking more about it, I would prefer security focused functional tests instead of unit tests on a higher priority. I like the idea that tests fail to begin with and students make them pass as they fix the vulnerabilities. To assist, we can add a branch with all tests passing for a reference.

[binarymist]

This is insane, I was just thinking about this as I was lying in bed and wondering about this exact issue, had forgotten that I had submitted it. That's a 10:4, Adding a branch with "all tests passing"... I'll work toward doing this as my Fascicle 1 gets near to handing to the technical editor, still a few months off.

[ckarande]

Thanks @binarymist . Assigning it to you. Great to know progress on Fascicle 1 :)

[mostafahussein]

Is there anything that I can do to complete this ? Let me know if you need any help

[binarymist]

Adding a branch with "all tests passing".

Means it has to be kept up to date as changes are made to master, this would need to be well documented, it'd also mean more work for contributors and mergers. Thoughts: @ckarande @lirantal @mostafahussein ?

[lirantal]

That's a great idea. Would love to see this, and as you said we should recognize the effort that this brings, especially with creating a baseline of tests for what we have so far.

[binarymist]

I don't think we have any tests other than the security regression test, which I'll be removing at some point... As soon as purpleteam is in prime-time, as purpleteam is using NodeGoat currently as the main SUT.

[mostafahussein]

We can break it down to phases or stages that needs to be done in the pipeline. So what about the following:

• Using TravisCI
• ESlint for checking code style (I know it is not a big deal at this point but we might use it at least to guarantee that any contributor follows the same code style )
• for the security checks i noticed that you already have snyk.io (this according to an issue that was opened before), am i right ? what else can we add ? maybe npm audit ?
  And sure if there are any kind of unit testing will be added too.

Lastly OWASP as in organization - or any other member responsible for this - might submit a proposal to Docker so we can have this as an official image which will allow us to have the automated security checks provided by dockerhub.

[binarymist]

Using TravisCI

Didn't you have something like this setup at some point @ckarande?

ESlint for checking code style (I know it is not a big deal at this point but we might use it at least to guarantee that any contributor follows the same code style )

NodeGoat currently uses "jsbeautifier" and "jshint" via npm script precommit, and contributors are expected to follow the guidelines documented. I personally have no issue swapping the grunt task with ESLint providing the same linting is happening, I think we need to consult @ckarande on this though.

I don't personally see a snyk.io report, not sure if @ckarande has this setup, keep in mind though, that the application is supposed to be insecure ;-)

I'm not sure what you would add for npm audit?

As @ckarande mentioned above:

security focused functional tests instead of unit tests on a higher priority

So we probably want to stick with that direction

[mostafahussein]

Okay, but I need clarification regarding this point: security focused functional tests, what kind of security tests do we need ? as you said it should be insecure 🐱
Forget about npm audit this is similar to synk, i mean it will check if there is any vulnerable packages and so on (NSP replacement) and actually we don't need these kind of security checks in here 😹

Do you mean by security focused functional tests where we have a script written in NodeJS that ensures that the main vulnerabilities of NodeGoat project exist ?

[binarymist]

Should this issue still even be open @ckarande ?

[ckarande]

@binarymist I would prefer to keep this issue open for any contributors to pick up or we implement it in future.

@mostafahussein thanks for your interest in it. About the security focused tests, the intention is to create a set of self-guided exercises that users can follow to fix the vulnerabilities in the NodeGoat app and get a feedback as they work on it. Here are some options to implement it:

• At a very basic code level, add failing tests written in jest or jasmine and mission for a user would be to make them pass by fixing the insecure code. For example, here is tutorial that teaches Rx API provided by Reactor, with the broken code, tests, and the solution branch. These tests are then also used to build a techio playground for those who prefer browser based UI instead of running tests on a console. However, this could be difficult implement right as there would be multiple ways to code a fix for a vulnerability. Also as @binarymist indicated, it would add more maintenance work for contributors and maintainers.
• The second option is to write browser based E2E tests using cypress or selenium, that would validate the DOM contents to pass or fail the tests. This is likely to be be more stable than the code level tests.
• Integrate the E2E tests with ZAP API scan to extend the test suite to wider set of vulnerabilities exploitable by ZAP.

I hope above helps. @binarymist @lirantal please suggest if you have any opinion or alternate suggestions on approaching this.

@binarymist, pretty impressive work with purpleteam project, please suggest if any of your work could be used as a reference for the purpose of these tests.

[binarymist]

This sounds very much like security regression testing, which we had a PoC for (still do, but is very much redundant with purpleteam). It was my intention to rip that PoC out, once I have purpleteam running in the Cloud... which shouldn't be far away now, although it looks like I may be taking another full time job soon.

purpleteam removes the need for any sort of test suite. The SaaS and CLI is smart enough to work out "how" to test based on "what" you tell it to test, which is provided by a build user config file. This one specifically targets NodeGoat as the SUT.

[ckarande]

Sounds good. Lets revisit this once you have the purpleteam running in the cloud. I would be very interested in seeing its demo with Nodegoat if possible.

[ckarande]

Addressed in #147