Technical_Documentation (old)

Technical Documentation

Modules

views.py

Central file to accept and propagate all requests from clients. Those include:

• upload firmware: User either drag and drops file or selects it by hand. For further information see (### file uploading)
  It is passed as FirmwareFile attached to the request which is stored as model in the database and saved packed on the server machine.
• analyse firmware: User fills django form which is send togehter with the request.
  It includes Reference to FirmwareFile, normal emba Options and expert mode Options, which is saved into the DB.
  Both the File and the Metainformation are passed to the BoundedExecuter (see ###)
• service dashboard: Displays currently running Analyzers with status information. Communication is done via asgi and channels. To retrieve the status information the emba log is read asynchronously
• central dashboard: TODO

file uploading

Archiver

The Archiver is basically an extended wrapper around shutils to support all common archive formats.
It supports packing, unpacking and validating of different archive types.
Unittest labeled as test_archiver.

BoundedExecutor

The BoundedExecutor is basically an extended wrapper around pythons ThreadPoolExecutor to support a selectable finite upper bound for running processes.
Internally it uses additional BoundedSemaphore to track the current load, on submit() the semaphore decrements on call.
On submit_firmware, the Archiver is utilized to unpack the selected firmware to an new directory and also starting the emba.sh process with the blocking subprocess.call.
Unittest labeled as test_boundedExecutor.

Frontend

• embaServiceDashboard.js: This script running on client side, takes the messages sent via websocket, to show live information about running emba processes in the backend. Currently, it displays the percentage, the current module as well as the current phase, the emba process is currently in. In the future, it shows a container per process with the previous information extended by the firmware name and some other details about the backend procedures.

• alertBox.js: This script provides alert functionalities to display success and error alerts to user. These functions can be used across the project when there is something to notify for the user.

• accumulatedReport.js: This script generates reports from the data collected on analyzing the firmware . These reports will be displayed in the main dashboard.

• fileUpload.js: This script allows the user to upload the firmware files and make post calls to backend to save files locally.

• main.js: This script contains functionalities related to navigation menu and also contains the delete firmware functionality .

• mainDashboard.js: This script generates the report of system load and also validates the login.

Backend

asgi

For live data exchange between server and client, the Django framework provides us with websocket communication via asgi, which is also called Django Channels. For server side handling the framework has the consumers.py class in place, whereas for client side handling you just have to open a websocket connection, opening the IP and Port specified in the backend. As a result, multiple clients can connect to the backend. The url routing is declared in the routing.py file, which is the equivalent to the urls.py file for HTTP communication.

log_reader.py

Our current server implementation creates a temporary empty log file and then waits in a blocking loop for new file changes on the emba.log via the inotify wrap. Whenever the log file has changed, the difference between the emba.log file and the temporary log file is calculated via a python difflib script and passed to a rxpy method for further processing. After extracting the relevant information from the emba.log, a temporary message dictionary is updated and appended to a global dictionary which contains all messages for all running processes. Whenever the global dictionary is updated, a message is sent via redis channel_layers to the WSConsumer class in consumers.py. Then the changes are written to the temporary log file so that future diff computations will be correct.

consumers.py

The WSConsumer class overrides the standard WebsocketConsumer implementation, so it fits our purposes. When a connection is initiated via the frontend, the class creates the room group for channels communication and then accepts the incoming websocket connection. The send_message method acts as a callback in the channel_layers communication and consequently is invoked whenever the Logreader class sends a message via channel_layers. The methos itself just extracts the dictionary, packs it as a json and passes it over to the frontend via the websocket.

inotify_wrap.py

The inotify reader class uses a python wrapper for the linux system call inotify(7). In our implementation, it adds a watch on the emba.log and sends events whenever the file changed. In this way we can trigger the next steps for live reading the emba.log.

Database

authentication

models

There are currently 2 different models:

• FirmwareFile:
  Used to track zipped firmwares. Featuring FileField as well as DateTimeField.
• Firmware:
  Used to track Analyzed firmwares. Custom fields provide additional attribute for defining expert field: CharFieldExpertMode, BooleanFieldExpertMode.
  Fields include reference to FirmwareFile, basic fields (e.g. version, vendor, architecture) and expert fields (e.g. web-report, multi-threading, dependencies).
  For further information on available flags consult the script like this: emba.sh -h

forms

The current forms are generated from above defined models. The forms additionally take care of html attributes like expert mode and class on instancing.
See https://docs.djangoproject.com/en/3.2/topics/forms/modelforms/ for further information.

Guidelines

testing

The project uses the django testing environment. To write your own unittest you need a python file labeled test_. Within there need to be class extending TestCase. On test execution all methodes in test classes are invoked an run.
For an example on unittests consult test_archiver and test_boundedExecutor.

There is Pipeline checking regression by running the django test environment: python manage.py test.
You are encouraged to run the tests locally beforehand.

logging

For logging use djangos logging environment.
Configuration can be found in embark/settings.py as dict LOGGING. Logs can be inspected at embark/logs/*.log

Usage (this will create the logfile 'web'):

logger = logging.getLogger('web')

[...]

logger.info("my very own log message")

For further reading see how to logging.

codestyle

Every contributor is obliged to the PEP8 coding style on python code. To check your conformity with pep8 locally: pycodestyle .
To get further information about the violation run: pycodestyle --show-source .
For further setting see pycodestyle documentation