Developer Blog¶
December 2021¶
GitLab CI/CD: In the rewrite I also extended the continuous integration and continuous deployment description from GitLab. More of the basic settings are described and a lot of examples are added. Also I give a better insight into analysis (security/performance/quality). And added a section to show how to simplify and make it equal in lots of repos with the use of templates. For the last step I also created my own template repository.
Update Site: As started last month with the stable use of mkdocs through the docker image image I also involuntarily stepped up to the newest versions of material for mkdocs and some of the extensions. This lead to a problem in tabs and a lot of newer possibilities.
I decided to first update the markdown writing guide with the new possibilities, the mkdocs configuration guide and will go over a lot of pages. Goal is to upstyle and update them all.
November 2021¶
Linux/Middleware: I added more information and help for sed, postgresql and some additions to the general Linux section.
Docker Mkdocs: After a lot of problems with mkdocs on different platforms and installations I switched completely to docker. Instead of a default image I made my own special docker image alinex/mkdocs. Find the sources at GitLab from which it is build. I checked that it can be used locally to build the documentation or within a build step. Also find my docker compendium for how to work with it.
Further I updated and extended the mkdocs package to support more diagrams, synced tabs, folder based navigation, update date and HTML minifier. And the documentation was restructured to better support development and IT operations.
Svelte: I am going further on with my tests. As far as I am I can say I like it. The performance is not as good as SolidJS but I like the structure, simplicity and tooling more here. It is also more complete but anyway new and progressing. Now as Going on I switched to the relatively new SvelteKit Framework with SSR support. To build my own components I plan to build them on top of SMUI (good but lacks documentation). I see the possibility to use it for static sites and also dynamic sites with an REST API based on feathersJS or Rust Warp.
Software Installer: Also using the Operations Toolkit I am building a customer specific software installer. It will install packages or manually install the software, copy configurations or change them so that a customer specific installation is done fully automatically on one or multiple hosts. It can work with different OS versions and can also analyze, update or remove all the software. We use this at work to manage over 200 virtual machines.
October 2021¶
Svelte: As a result to my previous experience with Rust aiming for the frontend I decided to look for a solution in JavaScript. My plan is to have a client framework which is powerful, responsive, with widgets and SSR support. A modern app should be server Rendered for fast loading and then hydrate into client control. Because of this the real frontend has to consist of a NodeJS Server and WebClient, a pure static server can't do this. But the business logic should be kept in other servers reachable from this and can be written in other languages.
After I learned Angular and React basics and used Vue for some time I now want to get a deeper look in Svelte. This newer framework directly compiles into clean HTML/CSS/JS and won't need a virtual DOM. The server part SvelteKit (like NextJS for React or Nuxt for Vue) is just under early development. But I want to check it out and started working on a sample project with it.
Data Checker: I made a specific data element checker which will check a specifically given object through all databases, filesystem and queues. Any problem or special finding will be reported and as far as possible automatic repair is suggested and can be applied by allowing it. This is also made with my Operations Toolkit.
Rust Server: I made a new attempt to build an Rust Server now switching to Warp as framework with the Tokio engine. But I just got it running to server a static site on minimal ressources. But as I further searched for Frontend Frameworks and looked over Yew which is based on WASM. But I got the impression that at the moment there is no perfect client framework directly coupled with Rust which shows a great prediction for the future.
Discovery: I could finish a core element in IT operations which will scan the network, find all hosts, ssh onto them and fully analyze them. It's made with my Operations Toolkit. The results are possible to shown on CLI, stored in database for further analysis and updated in our confluence pages per each host. Further a changelog is written and big changes can be pushed by a chatbot to mattermost channels.
September 2021¶
Operations Toolkit: I made a unique framework (closed source) which allows to create powerful it administration tools with ease. It contains a skeleton for applications with modules for command line interactions, output, logging, email, chatbot with a postgres base in TypeScript. To do the real actions a bash actor system was build which integrates all the linux cli possibilities locally and remote as easy to call methods into the TypeScript library.
Rust: Planning what to do using Rust. I will learn and try more in the next months and go again in the direction of a backend server. Later I plan to also try to create a web frontend using WASM. But at the moment it should not be for a specific application, only for a general framework with no current use case.
August 2021¶
Prometheus/Grafana: I set up both systems to be productively used. Prometheus will collect data from os, nginx-logs, jvm apps, tomcat, postgres, sql queries and use rules to precalculate something. Within Grafana general boards with variables and theme specific dashboards were created, too.
July 2021¶
Inventory: Based on the IT Operations Management Framework a discovery application will visit hosts, analyze them and collect the results in a postgres database. This information can be used to analyze the whole cluster of machines and to generate documentation out of it. IT will collect os, spu, memory, disk, mount, network, packages, software, config, services, users, ssh, cron and timers.
Data Analyzation: As a second project I made a specific data analyzation which is used to find problems in data structures by reading multiple postgres databases and the filesystem. This is used to find problems there processes won't work correctly on a record basis.
June 2021¶
Operations Framework: As designed in my plan for a IT Operations Management Framework I made the first Version running and implemented some applications. The framework currently based on TypeScript and Bash has the ability to interact with the user on a console, display information tables, log everything to file, to Email and send Messages to a Mattermost Chat System. First applications are a simple disk free check using df and du to find why the disk is full and the ability to delete something. Another one is a database check, which will detect problems in custom data elements like something has the wrong status or is not correctly processed.
Prometheus: I set up a test system with Prometheus + Grafana to collect system, postgres database and tomcat server metrics for a start. For visualization I also set up a Grafana server with specific Dashboards for all the Exporters. I will extend this to all systems used at my paid job. I aim to use this in production, too.
Back to Rust: It was 26 months ago as I last programmed Rust. I skipped it to be faster in creating solutions but now I plan to take a new approach. Over all this times I never lost Rust and followed the news. It gains more and more popularity and now it will be used partly in the Linux Kernel and also Google and Amazon are using it. That all shows it is the right way to go. So at first I will recheck my knowledge on the language itself, make some smaller applications (for test) and then go on planning productive systems in the second half of the year.
May 2021¶
IT Operations: That's what I mostly do, so why not bring together all that I learned so far and describe an architecture to monitor middle to big IT systems with thousands of separate services, processes and batches on hundreds of virtual machines in a propper way. A opinionated selection of applications is used here, Read more...
Deno: As this alternative implementation to NodeJS is growing I took my time to have a deeper look at it. Deno directly interprets TypeScript with an alternate module system. As I already use TypeScript in nearly all modules it looks great, but is incompatible for dual use with NodeJS of the same code in a lot of cases. So I will have to look again in a year and will decide if I switch to Deno or NodeJS with ESM.
JS Module System: As I started to upgrade my code again I got in conflict with common JS vs ES2020 module system. That is a complex battlefield, especially with typescript, mocha & Co involved. Therefore I upgraded my TypeScript Docs with special setups for the target environment which work.
My decision was to go up to ES2019 which is used on NodeJS 12 because NodeJS 10 reached it's end of life. But at this step I will keep on with CommonJS Modules. In about a year I will go up to ES2020 supporting NodeJS 14 and up, with the new ES modules.
Maintenance: This month I will work through all modules and update them, upgrade to node >=12.0, remove security problems, update used packages, check test coverage and maybe add some smaller extensions and fixes.
April 2021¶
Lots of fixes: As always if a big system with nearly 10k lines of code is ready to first be used, a big test phase is needed. Within this test phase I let it work, update the documentation and find a lot of bugs. This month I could fix over 50 bugs, problems and missing functionality. So this month was more a time to harden the checkup system with it's server component.
Checkup release: I could bring the next Version 1.3.0 ready now including all the postgres tests. Some general fixes were done, too. Currently it is running on a test system using the scheduler and database store through the server. But with this release I want to bring it further to production use. The ideas reach from only monitoring data over time, use it to check after deploy, on service startup and also within the loadbalancer.
March 2021¶
Postgres Analyzation: For the checkup system I added database tests to analyze simple database contents and check the postgres statistics of the cluster, database, schema or table in the Database Tests. Most information is read from the statistics tables, but as some information like inserts are absolute numbers since last reset I decided to use two values within a short time range to analyze the current changes per second. All data can be stored but keep in mind especially if you will use the postgres group to add the cluster with all elements below it will gather a lot of data in each run.
Adding Swap: To use suspend to disk I had to add a swap space while no partition or space for a new one is available. So I made a swap file in the existing partition. See my Manjaro installation page for the details.
February 2021¶
Log Analyzation: I got the concept of log analyzation completed for the next release which will come in a basic version soon. The analyzation is splitted into three types. Event analyzation will check if or how often a specific event occurred in the given time range, progress will check the time of a specific event and data will check an extracted value from the log. In all of them I will use a flexible read from local or remote systems, event parsing, generalization, filtering and validation.
Checkup for Production: The checkup module in it's current form will go productive now. It should work as scheduler in the server using some linux tests. Reporting will be made into database with data analyzation through DbVisualizer or DBeaver. But there are also some bugs which will be fixed within this step.
January 2021¶
Scheduler for Checkup: An integrated scheduler was planned to be independent of an external trigger like the linux cron daemon. The integrated scheduler works each run within the same process, therefore supports caching and if called through server result reuse. The configuration could be made very simple and directly in the format of a linux cron tab. If scheduler is used on CLI it will run as a daemon but in the server it will work as a background process. At the moment the results will go into the server log or if configured into datavase store. See Checkup Scheduler for more information.
Rollout of new Doc Theme: I did test multiple solutions to also create pdf and found that mkdocs-with-pdf is the only one working and giving nice looking results (previously I used mkdocs-mk2pdf-plugin which won't work on gitlab). See it for yourself on the PDF of this site. I updated the mkdocs description to show how to setup it with this. Further I optimized the style of the dark theme and added math formula support to mkdocs. I will do the rollout together with some minor updates on all modules.