Merge branch 'cpp_tests_update'

Update C++ tests

Add safe C++ test for simple connection test, device agnostic.
Better describe the use case of C++ tests in the Readme.

Related: https://github.com/d-e-s-o/nitrocli/issues/39

4 files changed, 176 insertions, 8 deletions

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 32f75a3..1004baf 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -211,6 +211,7 @@ IF (COMPILE_TESTS)
                 unittest/test_issues.cc
                 unittest/test_multiple_devices.cc
                 unittest/test_strdup.cpp
+                unittest/test_safe.cpp
         )
 
     foreach(testsourcefile ${TESTS} )
diff --git a/README.md b/README.md
index b101114..0b6b8d9 100644
--- a/README.md
+++ b/README.md
@@ -44,7 +44,7 @@ qmake ..
 make -j2
 ```
 
-### Windows MS Visual Studio 2017
+### Windows and Visual Studio 2017
 Lately Visual Studio has started handling CMake files directly. After opening the project's directory it should recognize it and initialize build system. Afterwards please run:
 1. `CMake -> Cache -> View Cache CMakeLists.txt -> CMakeLists.txt` to edit settings
 2. `CMake -> Build All` to build
@@ -167,21 +167,27 @@ The documentation of C API is included in the sources (could be  generated with
 Please check [NK_C_API.h](NK_C_API.h) (C API) for high level commands and [include/NitrokeyManager.h](include/NitrokeyManager.h) (C++ API). All devices' commands are listed along with packet format in [include/stick10_commands.h](include/stick10_commands.h) and [include/stick20_commands.h](include/stick20_commands.h) respectively for Nitrokey Pro and Nitrokey Storage products.
 
 # Tests
-Warning! Before you run unittests please either change both your Admin and User PINs on your Nitrostick to defaults (`12345678` and `123456` respectively) or change the values in tests source code. If you do not change them the tests might lock your device and lose your data. If it's too late, you can reset your Nitrokey using instructions from [homepage](https://www.nitrokey.com/de/documentation/how-reset-nitrokey).
+**Warning!** Most of the tests will overwrite user data. The only user-data safe tests are specified in `unittest/test_safe.cpp` (see *C++ tests* chapter).
+
+**Warning!** Before you run unittests please change both your Admin and User PINs on your Nitrostick to defaults (`12345678` and `123456` respectively), or change the values in tests source code. If you do not change them, the tests might lock your device temporarily. If it's too late already, you can reset your Nitrokey using instructions from [homepage](https://www.nitrokey.com/de/documentation/how-reset-nitrokey).
 
 ## Python tests
-libnitrokey has a great suite of tests written in Python 3 under the path: `unittest/test_*.py`: 
+libnitrokey has a great suite of tests written in Python 3 under the path: [unittest/test_*.py](https://github.com/Nitrokey/libnitrokey/tree/master/unittest): 
 * `test_pro.py` - contains tests of OTP, Password Safe and PIN control functionality. Could be run on both Pro and Storage devices.
 * `test_storage.py` - contains tests of Encrypted Volumes functionality. Could be run only on Storage.
+
 The tests themselves show how to handle common requests to device.
 Before running please install all required libraries with:
 ```bash
 cd unittest
 pip install --user -r requirements.txt
 ```
+or use Python's environment managing tool like [pipenv](https://pipenv.readthedocs.io/en/latest/) or `virtualenv`.
+
+
 To run them please execute: 
 ```bash
-# substitute <dev> with either pro or storage
+# substitute <dev> with either 'pro' or 'storage'
 py.test -v test_<dev>.py
 # more specific use - run tests containing in name <test_name> 5 times:
 py.test -v test_<dev>.py -k <test_name> --count 5
@@ -190,9 +196,76 @@ py.test -v test_<dev>.py -k <test_name> --count 5
 For additional documentation please check the following for [py.test installation](http://doc.pytest.org/en/latest/getting-started.html). For better coverage [randomly plugin](https://pypi.python.org/pypi/pytest-randomly) is installed - it randomizes the test order allowing to detect unseen dependencies between the tests.
 
 ## C++ tests
-There are also some unit tests implemented in C++, placed in unittest directory. They are not written as extensively as Python tests and are rather more a C++ low level interface check, often not using C++ API from `NitrokeyManager.cc`. Some of them are: [test_HOTP.cc](https://github.com/Nitrokey/libnitrokey/blob/master/unittest/test_HOTP.cc),
-[test.cc](https://github.com/Nitrokey/libnitrokey/blob/master/unittest/test.cc).
-Unit tests were written and tested on Ubuntu 16.04/16.10/17.04. To run them just execute binaries built in ./libnitrokey/build dir after enabling them by passing `-DCOMPILE_TESTS=ON` option like in `cmake .. -DCOMPILE_TESTS=ON && make`. 
+There are also some unit tests implemented in C++, placed in unittest directory. The only user-data safe online test set here is [test_safe.cpp](https://github.com/Nitrokey/libnitrokey/blob/master/unittest/test_safe.cpp), which tries to connect to the device, and collect its status data. Example run for Storage:
+```text
+# Storage device inserted, firmware version v0.53
+$ ./test_safe
+[Wed Jan  2 13:31:17 2019][DEBUG_L1]    => GET_DEVICE_STATUS
+..
+[Wed Jan  2 13:31:17 2019][DEBUG_L1]    <= GET_DEVICE_STATUS 0 1
+[Wed Jan  2 13:31:17 2019][DEBUG_L1]    => GET_PASSWORD_RETRY_COUNT
+[Wed Jan  2 13:31:17 2019][DEBUG_L1]    <= GET_PASSWORD_RETRY_COUNT 0 0
+[Wed Jan  2 13:31:17 2019][DEBUG_L1]    => GET_DEVICE_STATUS
+..
+[Wed Jan  2 13:31:17 2019][DEBUG_L1]    <= GET_DEVICE_STATUS 0 1
+[Wed Jan  2 13:31:17 2019][DEBUG_L1]    => GET_USER_PASSWORD_RETRY_COUNT
+[Wed Jan  2 13:31:17 2019][DEBUG_L1]    <= GET_USER_PASSWORD_RETRY_COUNT 0 0
+[Wed Jan  2 13:31:17 2019][DEBUG_L1]    => GET_DEVICE_STATUS
+...
+[Wed Jan  2 13:31:17 2019][DEBUG_L1]    <= GET_DEVICE_STATUS 0 1
+ transmission_data.dissect():   _padding:
+0000    00 00 00 00 00 00 00 00 00 00 00 00 00 05 2e 01   ................
+0010    00 00 -- -- -- -- -- -- -- -- -- -- -- -- -- --   ..
+ (int) SendCounter_u8:  0
+ (int) SendDataType_u8: 3
+ (int) FollowBytesFlag_u8:      0
+ (int) SendSize_u8:     28
+
+ MagicNumber_StickConfig_u16:   13080
+ (int) ReadWriteFlagUncryptedVolume_u8: 1
+ (int) ReadWriteFlagCryptedVolume_u8:   0
+ (int) ReadWriteFlagHiddenVolume_u8:    0
+ (int) versionInfo.major:       0
+ (int) versionInfo.minor:       53
+ (int) versionInfo.build_iteration:     0
+ (int) FirmwareLocked_u8:       0
+ (int) NewSDCardFound_u8:       1
+ (int) NewSDCardFound_st.NewCard:       1
+ (int) NewSDCardFound_st.Counter:       0
+ (int) SDFillWithRandomChars_u8:        1
+ ActiveSD_CardID_u32:   3670817656
+ (int) VolumeActiceFlag_u8:     1
+ (int) VolumeActiceFlag_st.unencrypted: 1
+ (int) VolumeActiceFlag_st.encrypted:   0
+ (int) VolumeActiceFlag_st.hidden:      0
+ (int) NewSmartCardFound_u8:    0
+ (int) UserPwRetryCount:        3
+ (int) AdminPwRetryCount:       3
+ ActiveSmartCardID_u32: 24122
+ (int) StickKeysNotInitiated:   0
+
+[Wed Jan  2 13:31:17 2019][DEBUG_L1]    => GET_DEVICE_STATUS
+..
+[Wed Jan  2 13:31:17 2019][DEBUG_L1]    <= GET_DEVICE_STATUS 0 1
+00005e3a
+[Wed Jan  2 13:31:17 2019][DEBUG_L1]    => GET_DEVICE_STATUS
+....
+[Wed Jan  2 13:31:18 2019][DEBUG_L1]    <= GET_DEVICE_STATUS 0 1
+[Wed Jan  2 13:31:18 2019][DEBUG_L1]    => GET_DEVICE_STATUS
+...
+[Wed Jan  2 13:31:18 2019][DEBUG_L1]    <= GET_DEVICE_STATUS 0 1
+===============================================================================
+All tests passed (18 assertions in 6 test cases)
+```
+Test's execution configuration and verbosity could be manipulated - please see `./test_safe --help` for details.
+
+The other tests sets are not written as extensively as Python tests and are rather more a C++ low level interface check used during the library development, using either low-level components, C API from `NK_C_API.cc`, or C++ API from `NitrokeyManager.cc`. Some of them are: [test_HOTP.cc](https://github.com/Nitrokey/libnitrokey/blob/master/unittest/test_HOTP.cc),
+[test1.cc](https://github.com/Nitrokey/libnitrokey/blob/master/unittest/test1.cc). See more in [unittest](https://github.com/Nitrokey/libnitrokey/tree/master/unittest) directory.
+
+**Note: these are not device model agnostic, and will most probably destroy your data on the device.**
+
+
+Unit tests were checked on Ubuntu 16.04/16.10/17.04. To run them just execute binaries built in `./libnitrokey/build` dir, after enabling them by passing `-DCOMPILE_TESTS=ON` option to `cmake` - e.g.: `cmake .. -DCOMPILE_TESTS=ON && make`. 
 
 
 The documentation of how it works could be found in nitrokey-app project's README on Github:
@@ -206,7 +279,7 @@ firmware code should show how things works:
 
 # Known issues / tasks
 * Currently only one device can be connected at a time (experimental work could be found in `wip-multiple_devices` branch),
-* C++ API needs some reorganization to C++ objects (instead of pointers to arrays). This will be also preparing for integration with Pybind11,
+* C++ API needs some reorganization to C++ objects (instead of pointers to byte arrays). This will be also preparing for integration with Pybind11,
 * Fix compilation warnings.
 
 Other tasks might be listed either in [TODO](TODO) file or on project's issues page.
diff --git a/unittest/test_offline.cc b/unittest/test_offline.cc
index 75bd316..279c38f 100644
--- a/unittest/test_offline.cc
+++ b/unittest/test_offline.cc
@@ -176,6 +176,7 @@ TEST_CASE("Test version getter", "[fast]") {
   // of commits since the last tag, and <hash> is the hash of the current
   // commit.  (This assumes that all tags have the name v<major>.<minor>.).
   // Optional <patch> field is allowed as well.
+  INFO("This test will fail, if the full git commit version was not collected during library build.");
   std::string s = library_version;
   std::string version("v[0-9]+\\.[0-9]+(\\.[0-9]+)?");
   std::string git_suffix("-[0-9]+-g[0-9a-z]+");
diff --git a/unittest/test_safe.cpp b/unittest/test_safe.cpp
new file mode 100644
index 0000000..d6f8b63
--- /dev/null
+++ b/unittest/test_safe.cpp
@@ -0,0 +1,93 @@
+/*
+ * Copyright (c) 2019 Nitrokey UG
+ *
+ * This file is part of libnitrokey.
+ *
+ * libnitrokey is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * any later version.
+ *
+ * libnitrokey is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public License
+ * along with libnitrokey. If not, see <http://www.gnu.org/licenses/>.
+ *
+ * SPDX-License-Identifier: LGPL-3.0
+ */
+
+#include "catch2/catch.hpp"
+
+#include <iostream>
+#include <cstring>
+#include "log.h"
+#include "../NK_C_API.h"
+
+int login;
+
+TEST_CASE("C API connect", "[BASIC]") {
+    INFO("This test set assumes either Pro or Storage device is connected.");
+    INFO("Here should be implemented only tests not changing the device's state, "
+         "and safe to user data.");
+
+    login = NK_login_auto();
+    REQUIRE(login != 0);
+    NK_set_debug_level(3);
+}
+
+TEST_CASE("Check retry count", "[BASIC]") {
+    INFO("This test assumes your PINs' attempt counters are set to 3");
+    REQUIRE(login != 0);
+    REQUIRE(NK_get_admin_retry_count() == 3);
+    REQUIRE(NK_get_user_retry_count() == 3);
+}
+
+void validate_cstring(char *s){
+    constexpr uint16_t max_length = 8*1024;
+    REQUIRE(s != nullptr);
+    REQUIRE(strnlen(s, max_length) > 0);
+    REQUIRE(strnlen(s, max_length) < max_length);
+    std::cout << s << std::endl;
+}
+
+TEST_CASE("Status command for Pro or Storage", "[BASIC]") {
+    REQUIRE(login != 0);
+    char* s = nullptr;
+
+    auto const m = NK_get_device_model();
+    REQUIRE(m != NK_DISCONNECTED);
+    if (m == NK_PRO)
+        s = NK_status();
+    else if (m == NK_STORAGE){
+        s = NK_get_status_storage_as_string();
+    }
+
+    validate_cstring(s);
+    free(s);
+    s = nullptr;
+}
+
+TEST_CASE("Device serial", "[BASIC]") {
+    REQUIRE(login != 0);
+    char* s = nullptr;
+    s = NK_device_serial_number();
+    validate_cstring(s);
+    free(s);
+    s = nullptr;
+}
+
+TEST_CASE("Firmware version", "[BASIC]") {
+    REQUIRE(login != 0);
+    // Currently all devices has major version '0'
+    // No firmware ever had a minor equal to '0'
+    REQUIRE(NK_get_major_firmware_version() == 0);
+    REQUIRE(NK_get_minor_firmware_version() != 0);
+}
+
+TEST_CASE("Library version", "[BASIC]") {
+    REQUIRE(NK_get_major_library_version() == 3);
+    REQUIRE(NK_get_minor_library_version() >= 4);
+}