Mike Gerwitz

Activist for User Freedom

aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorMike Gerwitz <mike.gerwitz@rtspecialty.com>2018-04-10 15:02:32 -0400
committerMike Gerwitz <mike.gerwitz@rtspecialty.com>2018-04-10 15:56:38 -0400
commit42d192af792f5890a6d707caee5cb3c910a5eb9b (patch)
tree2b886313d8cf6798169ee46cf0c0b2eae50a2f6f /progtest
parent8a17d0c6c575be59ace4d6b45fc59f972058a0ea (diff)
downloadtame-42d192af792f5890a6d707caee5cb3c910a5eb9b.tar.gz
tame-42d192af792f5890a6d707caee5cb3c910a5eb9b.tar.bz2
tame-42d192af792f5890a6d707caee5cb3c910a5eb9b.zip
progtest: Exit with non-zero status on test failure
Not a very useful test runner if it doesn't ever fail, now is it? * Makefile.am (check): Invoke new test/runner-test. Depend on modindex. * bin/runner.js: Exit with non-zero status on assertion failure. * test/_stub: Add stub program with good and bad test cases to test exit code. * test/runner-test: Add system test.
Diffstat (limited to 'progtest')
-rw-r--r--progtest/Makefile.am3
-rw-r--r--progtest/bin/runner.js15
-rw-r--r--progtest/test/_stub/bad.yml7
-rw-r--r--progtest/test/_stub/good.yml7
-rw-r--r--progtest/test/_stub/program.js25
-rwxr-xr-xprogtest/test/runner-test40
6 files changed, 95 insertions, 2 deletions
diff --git a/progtest/Makefile.am b/progtest/Makefile.am
index 4276795..9af1d9c 100644
--- a/progtest/Makefile.am
+++ b/progtest/Makefile.am
@@ -33,9 +33,10 @@ modindex: $(nsindex)
$(CURDIR)/build-aux/gen-index "$*" > "$@"
test: check
-check:
+check: modindex
PATH="$(PATH):$(CURDIR)/node_modules/mocha/bin" \
mocha @NODE_DESTRUCTURE@ --recursive test/
+ test/runner-test
browserify: tame-progtest.js
tame-progtest.js: check modindex
diff --git a/progtest/bin/runner.js b/progtest/bin/runner.js
index c6c5deb..4dddf20 100644
--- a/progtest/bin/runner.js
+++ b/progtest/bin/runner.js
@@ -33,4 +33,17 @@ const runner = require( '../src/env' ).console(
);
runner( case_yaml )
- .catch( e => console.error( e ) );
+ .then( results =>
+ {
+ const failed = results.some(
+ ( { failures } ) => failures.length
+ );
+
+ process.exit( +failed );
+ } )
+ .catch( e =>
+ {
+ console.error( e );
+ process.exit( 1 );
+ } );
+
diff --git a/progtest/test/_stub/bad.yml b/progtest/test/_stub/bad.yml
new file mode 100644
index 0000000..793f0c9
--- /dev/null
+++ b/progtest/test/_stub/bad.yml
@@ -0,0 +1,7 @@
+# Simple test case that should fail
+
+- description: Failure
+ data:
+ in: 0
+ expect:
+ out: 1
diff --git a/progtest/test/_stub/good.yml b/progtest/test/_stub/good.yml
new file mode 100644
index 0000000..a4c682e
--- /dev/null
+++ b/progtest/test/_stub/good.yml
@@ -0,0 +1,7 @@
+# Simple test case that should succeed
+
+- description: Success
+ data:
+ in: 1
+ expect:
+ out: 1
diff --git a/progtest/test/_stub/program.js b/progtest/test/_stub/program.js
new file mode 100644
index 0000000..cb9e67d
--- /dev/null
+++ b/progtest/test/_stub/program.js
@@ -0,0 +1,25 @@
+/**
+ * Stub program ("rater")
+ *
+ * Copyright (C) 2018 R-T Specialty, LLC.
+ *
+ * This file is part of TAME.
+ *
+ * TAME is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+exports.rater = data =>
+{
+ return { vars: { out: data.in } };
+};
diff --git a/progtest/test/runner-test b/progtest/test/runner-test
new file mode 100755
index 0000000..5091bd3
--- /dev/null
+++ b/progtest/test/runner-test
@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+#
+# Runner script system test
+#
+# Copyright (C) 2018 R-T Specialty, LLC.
+#
+# This file is part of TAME.
+#
+# TAME is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+set -euo pipefail
+
+cd "$( dirname "$0" )"
+
+declare -r bin=../bin
+
+
+main()
+{
+ set -x
+
+ # should succeed
+ "$bin"/runner _stub/program.js _stub/good.yml >/dev/null
+
+ # should fail
+ "$bin"/runner _stub/program.js _stub/bad.yml >/dev/null && exit 1 || true
+}
+
+main "$@"