Documentation update

Signed-off-by: Szczepan Zalega <szczepan@nitrokey.com>

3 files changed, 126 insertions, 71 deletions

diff --git a/README.md b/README.md
index 79c36d7..99c4a44 100644
--- a/README.md
+++ b/README.md
@@ -1,12 +1,13 @@
 [![Stories in Ready](https://badge.waffle.io/Nitrokey/libnitrokey.png?label=ready&title=Ready)](https://waffle.io/Nitrokey/libnitrokey)
+[![Build Status](https://travis-ci.org/Nitrokey/libnitrokey.svg?branch=master)](https://travis-ci.org/Nitrokey/libnitrokey)
 # libnitrokey
-Libnitrokey is a project to communicate with Nitrokey stick devices in clean and easy manner. Written in C++14, testable with `Catch` framework, with C API, Python access (through CFFI and C API, in future with Pybind11).
+Libnitrokey is a project to communicate with Nitrokey Pro and Storage devices in a clean and easy manner. Written in C++14, testable with `py.test` and `Catch` frameworks, with C API, Python access (through CFFI and C API, in future with Pybind11).
 
-The development of this project is aimed to make it itself a living documentation of communication protocol between host and the Nitrokey stick devices.
+The development of this project is aimed to make it itself a living documentation of communication protocol between host and the Nitrokey stick devices. The command packets' format is described here: [Pro v0.7](include/stick10_commands.h), [Pro v0.8](include/stick10_commands_0.8.h), [Storage](include/stick20_commands.h). Handling and additional operations are described here: [NitrokeyManager.cc](NitrokeyManager.cc).
 
-A C++14 complying compiler is required. For feature support tables please check [table 1](https://gcc.gnu.org/projects/cxx-status.html#cxx14) or [table 2](http://en.cppreference.com/w/cpp/compiler_support).
+A C++14 complying compiler is required due to heavy use of variable templates. For feature support tables please check [table 1](https://gcc.gnu.org/projects/cxx-status.html#cxx14) or [table 2](http://en.cppreference.com/w/cpp/compiler_support).
 
-Libnitrokey is developed with the latest compilers: g++ 5.4+, clang 3.8+
+Libnitrokey is developed and tested with the latest compilers: g++ 6.2, clang 3.8. We use Travis CI to test builds also on g++ 5.4 and under OSX compilers starting up from xcode 6.4 environment (OSX 10.10). 
 
 ## Getting sources
 This repository uses `git submodules`.
@@ -22,24 +23,40 @@ git submodule update --init --recursive
 ```
 
 ## Dependencies
-Following packages are needed to use libnitrokey (Debian/Ubuntu):
+Following libraries are needed to use libnitrokey on Linux (names of the packages on Ubuntu):
 - libhidapi-dev [(HID API)](http://www.signal11.us/oss/hidapi/)
+- libusb-1.0-0-dev 
 
 
 ## Compilation
-To compile library using Clang please run `make`. If you have GCC and would like to use it instead you can run:
+libnitrokey uses CMake as its build system. To compile please run following sequence of commands:
 ```bash
- make CXX=g++
+# assuming current dir is ./libnitrokey/
+mkdir -p build
+cd build
+cmake .. <OPTIONS>
+make -j2
 ```
-This should create a library file under path build/libnitrokey.so and compile C++ tests in unittest/ directory.
 
-## Using with Python
-To use libnitrokey with Python a [CFFI](http://cffi.readthedocs.io/en/latest/overview.html) library is required (either 2.7+ or 3.0+). It can be installed with
+By default (with empty `<OPTIONS>` string) this will create two static library files - build/libnitrokey-static.a and build/libnitrokey-static-log.a. If you wish to build another version (e.g. shared library to use with Python) you can use as `<OPTIONS>` string `-DLIBNITROKEY_STATIC=OFF`. All options could be listed with `cmake .. -L` or instead `cmake` a `ccmake ..` tool could be used for configuration (where `..` is the path with `CMakeLists.txt` file). `ccmake` shows also description of the build parameters.
+
+If you have trouble compiling or running the library you can check [.travis.yml](.travis.yml) file for configuration details. This file is used by our CI service to make test builds on OSX and Ubuntu 14.04.
+
+Other build options (all take either `ON` or `OFF`):
+* ADD_ASAN - add tests for memory leaks and out-of-bounds access
+* ADD_TSAN - add tests for threads race, needs USE_CLANG
+* USE_CLANG - forces Clang as the compiler
+* COMPILE_TESTS - compile C++ tests
+
+
+
+# Using with Python
+To use libnitrokey with Python a [CFFI](http://cffi.readthedocs.io/en/latest/overview.html) library is required (either 2.7+ or 3.0+). It can be installed with:
 ```bash
-sudo pip install cffi # for python 2.x
+pip install --user cffi # for python 2.x
 pip3 install cffi # for python 3.x
 ```
-Just import it and read the C API header and it is done! You have access to the library. Example code printing HOTP code for both devices (assuming they are both connected):
+Just import it, read the C API header and it is done! You have access to the library. Here is an example printing HOTP code for Pro or Storage device, assuming it is run in root directory [(full example)](python_bindings_example.py):
 ```python
 #!/usr/bin/env python
 import cffi
@@ -51,20 +68,48 @@ get_string = ffi.string
 def get_library():
     fp = 'NK_C_API.h'  # path to C API header
 
-   declarations = []
+    declarations = []
     with open(fp, 'r') as f:
         declarations = f.readlines()
 
     a = iter(declarations)
     for declaration in a:
-        if declaration.startswith('extern') and not '"C"' in declaration:
-            declaration = declaration.replace('extern', '').strip()
+        if declaration.startswith('NK_C_API'):
+            declaration = declaration.replace('NK_C_API', '').strip()
             while not ';' in declaration:
                 declaration += (next(a)).strip()
-            # print(declaration)
-            ffi.cdef(declaration)
+            print(declaration)
+            ffi.cdef(declaration, override=True)
+
+    C = None
+    import os, sys
+    path_build = os.path.join(".", "build")
+    paths = [ os.path.join(path_build,"libnitrokey-log.so"),
+              os.path.join(path_build,"libnitrokey.so")]
+    for p in paths:
+        print p
+        if os.path.exists(p):
+            C = ffi.dlopen(p)
+            break
+        else:
+            print("File does not exist: " + p)
+            print("Trying another")
+    if not C:
+        print("No library file found")
+        sys.exit(1)
+
+    C.NK_set_debug(False)
+    nk_login = C.NK_login_auto() # try to connect firstly to Pro and then to Storage
+    if nk_login != 1:
+        print('No devices detected!')
+    assert nk_login != 0  # returns 0 if not connected or wrong model or 1 when connected
+    global device_type
+    firmware_version = C.NK_get_major_firmware_version()
+    model = 'P' if firmware_version in [7,8] else 'S'
+    device_type = (model, firmware_version)
+
+    C.NK_set_debug(True)
 
-    C = ffi.dlopen("build/libnitrokey.so")  # path to built library
     return C
 
 
@@ -73,63 +118,62 @@ def get_hotp_code(lib, i):
 
 
 libnitrokey = get_library()
-libnitrokey.NK_set_debug(False)  # do not show debug messages
-
-libnitrokey.NK_login('P')  # connect to Nitrokey Pro device
-hotp_slot_1_code = get_hotp_code(libnitrokey, 1)
-print('Getting HOTP code from Nitrokey Pro: ')
-print(hotp_slot_1_code)
-libnitrokey.NK_logout()  # disconnect device
+libnitrokey.NK_set_debug(False)  # do not show debug messages (log library only)
 
-libnitrokey.NK_login('S')  # connect to Nitrokey Storage device
-hotp_slot_1_code_nitrokey_storage = get_hotp_code(libnitrokey, 1)
-print('Getting HOTP code from Nitrokey Storage: ')
-print(hotp_slot_1_code_nitrokey_storage)
+hotp_slot_code = get_hotp_code(libnitrokey, 1)
+print('Getting HOTP code from Nitrokey device: ')
+print(hotp_slot_code)
 libnitrokey.NK_logout()  # disconnect device
 ```
-In case one of the devices or no devices are connected, unfriendly message will be printed.
-All available functions for C and Python are listed in NK_C_API.h. Please check `Documentation` section below.
+
+In case  no devices are connected, a friendly message will be printed.
+All available functions for C and Python are listed in [NK_C_API.h](NK_C_API.h). Please check `Documentation` section below.
 
 ## Documentation
 The documentation of C API is included in the sources (could be  generated with doxygen if requested).
 Please check NK_C_API.h (C API) for high level commands and include/NitrokeyManager.h (C++ API). All devices' commands are listed along with packet format in include/stick10_commands.h and include/stick20_commands.h respectively for Nitrokey Pro and Nitrokey Storage products.
 
-#Tests
-Warning! Before you run unittests please either change both your Admin and User PINs on your Nitrostick to defaults (`12345678` and `123456` respectively) or change the values in tests source code. If you do not change them the tests might lock your device. If its too late, you can always reset your Nitrokey using instructions from [homepage](https://www.nitrokey.com/de/documentation/how-reset-nitrokey).
+# Tests
+Warning! Before you run unittests please either change both your Admin and User PINs on your Nitrostick to defaults (`12345678` and `123456` respectively) or change the values in tests source code. If you do not change them the tests might lock your device. If it's too late, you can always reset your Nitrokey using instructions from [homepage](https://www.nitrokey.com/de/documentation/how-reset-nitrokey).
 
 ## Python tests
-Libnitrokey has a couple of tests written in Python under the path: `unittest/test_bindings.py`. The tests themselves show how to handle common requests to device.
-To run them please enter `unittest` directory and execute `py.test -v` (please check the following for [py.test installation](http://doc.pytest.org/en/latest/getting-started.html)). For even better coverage [randomly plugin](https://pypi.python.org/pypi/pytest-randomly) could be installed.
-
-## C++ tests
-There are also some unit tests implemented in C++:
-[test_HOTP.cc](https://github.com/Nitrokey/libnitrokey/blob/master/unittest/test_HOTP.cc) (passing)
-[test.cc](https://github.com/Nitrokey/libnitrokey/blob/master/unittest/test.cc) (not passing).
-Unit tests was written and tested with Nitrokey Pro on Ubuntu 16.04. To run them just execute binaries build in unittest/build dir after initial make. You will have to add LD_LIBRARY_PATH variable to environment though, like:
+Libnitrokey has a great suite of tests written in Python under the path: `unittest/test_*.py`: 
+* `test_pro.py` - contains tests of OTP, Password Safe and PIN control functionality. Could be run on both Pro and Storage devices.
+* `test_storage.py` - contains tests of Encrypted Volumes functionality. Could be run only on Storage.
+The tests themselves show how to handle common requests to device.
+Before running please install all required libraries with:
+```bash
+cd unittest
+pip install --user -r requirements.txt
+```
+To run them please execute: 
 ```bash
-cd unittests/build
-LD_LIBRARY_PATH=. ./test_HOTP
+# substitute <dev> with either pro or storage
+py.test -v test_<dev>.py
+# more specific use - run tests containing in name <test_name> 5 times:
+py.test -v test_<dev>.py -k <test_name> --count 5
+
 ```
-or just execute `./run.sh`.
-The device's commands are here:
-[stick10_commands.h](https://github.com/Nitrokey/libnitrokey/blob/hotp_tests/include/stick10_commands.h)
-for Nitrokey Pro and
-[stick20_commands.h](https://github.com/Nitrokey/libnitrokey/blob/hotp_tests/include/stick20_commands.h)
-for Nitrokey Storage [which includes Nitrokey Pro commands set].
+For additional documentation please check the following for [py.test installation](http://doc.pytest.org/en/latest/getting-started.html). For better coverage [randomly plugin](https://pypi.python.org/pypi/pytest-randomly) is installed - it randomizes the test order allowing to detect unseen dependencies between the tests.
+
+## C++ tests
+There are also some unit tests implemented in C++, placed in unittest directory. They are not written as extensively as Python tests and are rather more a C++ low level interface check, often not using C++ API from NitrokeyManager.cc. Some of them are: [test_HOTP.cc](https://github.com/Nitrokey/libnitrokey/blob/master/unittest/test_HOTP.cc)
+[test.cc](https://github.com/Nitrokey/libnitrokey/blob/master/unittest/test.cc)
+Unit tests were written and tested on Ubuntu 16.04/16.10. To run them just execute binaries built in ./libnitrokey/build dir after enabling them by passing `-DCOMPILE_TESTS` option like in `cmake .. -DCOMPILE_TESTS && make`. 
+
 
 The documentation of how it works could be found in nitrokey-app project's README on Github:
-[Nitrokey-app - internals](https://github.com/Nitrokey/nitrokey-app/blob/master/README.md#internals) - some typos might be in links there, but one can traverse to destination manually by looking at URL.
+[Nitrokey-app - internals](https://github.com/Nitrokey/nitrokey-app/blob/master/README.md#internals).
 
-To peek/debug communication with device running nitrokey-app in debug mode [-d switch] and checking the logs
-[right click on tray icon and then 'Debug'] might be helpful. Also crosschecking with
+To peek/debug communication with device running nitrokey-app (0.x branch) in debug mode (`-d` switch) and checking the logs
+(right click on tray icon and then 'Debug') might be helpful. Latest Nitrokey App (1.x branch) uses libnitrokey to communicate with device. Once linked with `libnitrokey-log` it will print all communication to the console. Additionally crosschecking with
 firmware code should show how things works:
 [report_protocol.c](https://github.com/Nitrokey/nitrokey-pro-firmware/blob/master/src/keyboard/report_protocol.c)
-[for Nitrokey Pro, for Storage similarly].
+(for Nitrokey Pro, for Storage similarly).
 
 # Known issues / tasks
 * Currently only one device can be connected at a time
 * C++ API needs some reorganization to C++ objects (instead of pointers to arrays). This will be also preparing for integration with Pybind11,
-* The library is not supporting Nitrokey Storage stick but it should be done in nearest future. The only working function for now (looking by Python unit tests) is getting HOTP code.
 * Fix compilation warnings
 
 Other tasks might be listed either in [TODO](TODO) file or on project's issues page.
diff --git a/TODO b/TODO
index 1eb99b4..3183a43 100644
--- a/TODO
+++ b/TODO
@@ -1,10 +1 @@
-
-OTP autorization
-cleanup in C API cc
 use strings instead of char* and vectors instead of others
-add tests for SLOT config options
-* fix tests so they could run in random order
-
-NK Storage:
-TOTP - check setting time function
-HOTP - counter is send via string, not number
diff --git a/python_bindings_example.py b/python_bindings_example.py
index 6bad8f4..37b70b8 100755
--- a/python_bindings_example.py
+++ b/python_bindings_example.py
@@ -4,7 +4,7 @@ from enum import Enum
 
 """
 This example will print 10 HOTP codes from just written HOTP#2 slot.
-For more examples of use please refer to unittest/test_bindings.py file.
+For more examples of use please refer to unittest/test_*.py files.
 """
 
 ffi = cffi.FFI()
@@ -27,20 +27,39 @@ def get_library():
 
     a = iter(declarations)
     for declaration in a:
-        if declaration.startswith('extern') and not '"C"' in declaration:
-            declaration = declaration.replace('extern', '').strip()
+        if declaration.startswith('NK_C_API'):
+            declaration = declaration.replace('NK_C_API', '').strip()
             while not ';' in declaration:
                 declaration += (next(a)).strip()
-            # print(declaration)
-            ffi.cdef(declaration)
+            #print(declaration)
+            ffi.cdef(declaration, override=True)
+
+    C = None
+    import os, sys
+    path_build = os.path.join(".", "build")
+    paths = [ os.path.join(path_build,"libnitrokey-log.so"),
+              os.path.join(path_build,"libnitrokey.so")]
+    for p in paths:
+        print p
+        if os.path.exists(p):
+            C = ffi.dlopen(p)
+            break
+        else:
+            print("File does not exist: " + p)
+            print("Trying another")
+    if not C:
+        print("No library file found")
+        sys.exit(1)
 
-    C = ffi.dlopen("build/libnitrokey.so")  # path to built library
     return C
 
 
 def get_hotp_code(lib, i):
     return lib.NK_get_hotp_code(i)
 
+def to_hex(ss):
+    return ''.join([ format(ord(s),'02x') for s in ss ])
+
 print('Warning!')
 print('This example will change your configuration on inserted stick and overwrite your HOTP#2 slot.')
 print('Please write "continue" to continue or any other string to quit')
@@ -52,11 +71,12 @@ if not a == 'continue':
 ADMIN = raw_input('Please enter your admin PIN (empty string uses 12345678)')
 ADMIN = ADMIN or '12345678'  # use default if empty string
 
+show_log = raw_input('Should log messages be shown (please write "yes" to enable)?') == 'yes'
 libnitrokey = get_library()
-libnitrokey.NK_set_debug(False)  # do not show debug messages
+libnitrokey.NK_set_debug(show_log)  # do not show debug messages
 
 ADMIN_TEMP = '123123123'
-RFC_SECRET = '12345678901234567890'
+RFC_SECRET = to_hex('12345678901234567890')
 
 # libnitrokey.NK_login('S')  # connect only to Nitrokey Storage device
 # libnitrokey.NK_login('P')  # connect only to Nitrokey Pro device