Leave a comment Posted on 17.11.2021 by 891rpm Без рубрики

How to Convert Images to WebP Format in Linux

One of the numerous best practices you will hear of, for optimizing your web-site performance is using compressed images. In this article, we will share with you a new image format called webp for creating compressed and quality images for the web.

WebP is a relatively new, open source image format that offers exceptional lossless and lossy compression for images on the web, designed by Google. To use it, you need to download pre-compiled utilities for Linux, Windows and Mac OS X.

With this modern image format, webmasters and web developers can create smaller, richer images that make the web faster.

How to Install WebP Tool in Linux

Thankfully, the webp package is present in the Ubuntu official repositories, you can install it using the APT package manager as shown.

$ sudo apt install webp

On other Linux distributions, start by downloading the webp package from Googles repository using the wget command as follows.

$ wget -c https://storage.googleapis.com/downloads.webmproject.org/releases/webp/libwebp-0.6.1-linux-x86-32.tar.gz

Now extract the archive file and move into the extracted package directory as follows.

$ tar -xvf libwebp-0.6.1-linux-x86-32.tar.gz 
$ cd libwebp-0.6.1-linux-x86-32/
$ cd bin/
$ ls

As you can see from the above screen shot, the package contains a precompiled library (libwebp) for adding webp encoding or decoding to your programs and various webp utilities listed below.

anim_diff – tool to display the difference between animation images.
anim_dump – tool to dump the difference between animation images.
cwebp – webp encoder tool.
dwebp – webp decoder tool.
gif2webp – tool for converting GIF images to webp.
img2webp – tools for converting a sequence of images into an animated webp file.
vwebp – webp file viewer.
webpinfo – used to view info about a webp image file.
webpmux – webp muxing tool.

To convert an image to webp, you can use the cwebp tool, where the -q switch defines the output quality and -o specifies the output file.

$ cwebp -q 60 Cute-Baby-Girl.png -o Cute-Baby-Girl.webp
OR
$ ./cwebp -q 60 Cute-Baby-Girl.png -o Cute-Baby-Girl.webp

You can view the converted webp image using the vwebp tool.

$ ./vwebp Cute-Baby-Girl.webp

You can see all options for any of the tools above by running them without any arguments or using the -longhelp flag, for example.

$ ./cwebp -longhelp

Last but not least, if you want to run the above programs without typing their absolute paths, add the directory ~/libwebp-0.6.1-linux-x86-32/bin to your PATH environmental variable in your ~/.bashrc file.

$ vi ~/.bashrc

Add the line below towards the end of the file.

export PATH=$PATH:~/libwebp-0.6.1-linux-x86-32/bin

Save the file and exit. Then open a new terminal window and you should be able to run all webp programs like any other system commands.

WebP Project Homepage: https://developers.google.com/speed/webp/

Leave a comment Posted on 02.11.2021 by 891rpm Без рубрики

Установка Redmine 4.0.5 NGINX PostgreSQL в Centos 7

Redmine — открытое серверное веб-приложение для управления проектами и задачами (в том числе для отслеживания ошибок). Redmine написан на Ruby и представляет собой приложение на основе широко известного веб-фреймворка Ruby on Rails

Цикл статей по установке и настройке Redmine

Содержание

Подготовительный этап

Устанавливаем пакеты, необходимые для сборки Redmine и Ruby из исходного кода

$ sudo yum install nano curl gpg gcc gcc-c++ make patch autoconf automake bison libffi-devel libtool libcurl-devel
$ sudo yum install readline-devel sqlite-devel zlib-devel openssl-devel readline  glibc-headers glibc-devel
$ sudo yum install zlib libyaml-devel bzip2 iconv-devel ImageMagick ImageMagick-devel gdbm-devel  mariadb-devel

Создадим нового пользователя, добавляем его в группу Wheel

$ sudo useradd -m -U -r -d /opt/redmine redmine
$ sudo usermod -aG wheel redmine
$ sudo chmod 750 /opt/redmine

Разрешаем пользователю redmine делать sudo, не запрашивая пароль

$ echo "redmine ALL=(ALL) NOPASSWD: ALL" | sudo tee -a /etc/sudoers.d/redmine > /dev/null

Добавляем правила в Firewall

$ sudo firewall-cmd --permanent --zone=public --add-service=http
$ sudo firewall-cmd --permanent --zone=public --add-service=https 
$ sudo firewall-cmd --reload
$ sudo firewall-cmd --list-all-zones

Установка и настройка PostgreSQL 10

Добавляем репозиторий

$ sudo yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm

Устанавливаем PostgreSQL

$ sudo yum install libpqxx libpqxx-devel postgresql10.x86_64 postgresql10-server postgresql10-contrib postgresql10-libs postgresql10-tcl postgresql10-devel protobuf-devel

Инициализируем пространство для БД

$ sudo /usr/pgsql-10/bin/postgresql-10-setup initdb

Добавляем службу в автозагрузку и запускаем PostgreSQL

$ sudo systemctl start postgresql-10
$ sudo systemctl enable postgresql-10

Переключаемся на пользователя postgres

$ sudo su - postgres

Создаем пользователя БД

-bash-4.2$ createuser redmine

Переключаемся в PostgreSQL shell

-bash-4.2$ psql

Задаем пароль для пользователя БД

postgres=# ALTER USER redmine WITH ENCRYPTED password '%passwd%';

Создам базу и задаем владельца базы

postgres=# CREATE DATABASE redmine WITH ENCODING='UTF8' OWNER=redmine;
Exit
\q
logout

Настраиваем доступ к PostgreSQL

$ sudo nano /var/lib/pgsql/10/data/pg_hba.conf
# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            md5
# IPv6 local connections:
host    all             all             ::1/128                 md5

Перезапускаем PostgreSQL

$ sudo systemctl restart postgresql-10

Установка Ruby из исходников

Переключаемся на пользователя redmine

$ sudo su - redmine

Скачиваем ruby 2.6.5, собираем из исходников

$ curl -L https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.5.tar.gz -o /tmp/ruby-2.6.5.tar.gz 
$ cd /tmp
$ tar zxf /tmp/ruby-2.6.5.tar.gz
$ sudo mv /tmp/ruby-2.6.5 /opt/ruby
$ sudo chown -R redmine. /opt/ruby
$ cd /opt/ruby
$ ./configure --disable-install-doc
$ make -j2
$ sudo make install

Что бы удалить собранный из исходников ruby:

$ sudo make clean

Проверяем

$ ruby -v
ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]

Установка Rubygems из исходников

Скачиваем rubygems 3.0.6, собираем из исходников

$ curl -L https://rubygems.org/rubygems/rubygems-3.0.6.tgz -o /tmp/rubygems-3.0.6.tgz
$ cd /tmp
$ tar -zxf rubygems-3.0.6.tgz
$ cd rubygems-3.0.6
$ sudo chown -R redmine. /usr/local/lib/ruby/
$ alias sudo='sudo env PATH=$PATH'
$ sudo ruby setup.rb

Проверяем

$ gem -v
3.0.6

Устанавливаем Redmine. Начало

Скачиваем и распаковываем redmine

$ cd /tmp
$ curl -L http://www.redmine.org/releases/redmine-4.0.5.tar.gz -o /tmp/redmine-4.0.5.tar.gz
$ tar zxf /tmp/redmine-4.0.5.tar.gz
$ sudo mv /tmp/redmine-4.0.5 /opt/redmine/redmine-4.0.5

Создаем симлинк

$ sudo ln -s /opt/redmine/redmine-4.0.5 /opt/redmine/redmine-latest
$ sudo chown -h redmine:redmine /opt/redmine/redmine-latest

Настраиваем подключение к PostgreSQL

$ cp /opt/redmine/redmine-4.0.5/config/database.yml.example /opt/redmine/redmine-4.0.5/config/database.yml
$ nano /opt/redmine/redmine-4.0.5/config/database.yml

production:
  adapter: postgresql
  database: redmine
  host: localhost
  username: redmine
  password: "%passwd%"
  encoding: utf8

Устанавливаем Redmine (если сервер имеет выход в интернет)

$ alias sudo='sudo env PATH=$PATH'
$ cd /opt/redmine/redmine-4.0.5
$ sudo gem install bundler

Если сервер не имеет выход в интернет, надо скачать необходимые пакеты (gem)

gem "bundler", ">= 1.5.0" https://rubygems.org/downloads/bundler-2.0.2.gem
gem "rails", "5.2.3" https://rubygems.org/downloads/rails-5.2.3.gem
gem "rouge", "~> 3.3.0" https://rubygems.org/downloads/rouge-3.3.0.gem
gem "request_store", "1.0.5" https://rubygems.org/downloads/request_store-1.0.5.gem
gem "mini_mime", "~> 1.0.1" https://rubygems.org/downloads/mini_mime-1.0.1.gem
gem "actionpack-xml_parser" https://rubygems.org/downloads/actionpack-xml_parser-2.0.1.gem
gem "roadie-rails", "~> 1.3.0" https://rubygems.org/downloads/roadie-rails-1.3.0.gem
gem "mimemagic" https://rubygems.org/downloads/mimemagic-0.3.3.gem
gem "mail", "~> 2.7.1" https://rubygems.org/downloads/mail-2.7.1.gem
gem "csv", "~> 3.0.1" if RUBY_VERSION >= "2.3" && RUBY_VERSION < "2.6" https://rubygems.org/downloads/csv-3.1.0.gem
gem "nokogiri", (RUBY_VERSION >= "2.3" ? "~> 1.10.0" : "~> 1.9.1") https://rubygems.org/downloads/nokogiri-1.10.5.gem
gem "i18n", "~> 0.7.0" https://rubygems.org/downloads/i18n-0.7.0.gem
gem "xpath", "< 3.2.0" if RUBY_VERSION < "2.3" https://rubygems.org/downloads/xpath-3.2.0.gem
gem "sprockets", "~> 3.7.2" https://rubygems.org/downloads/sprockets-3.7.2.gem
gem 'tzinfo-data', platforms: [:mingw, :x64_mingw, :mswin] https://rubygems.org/downloads/tzinfo-data-1.2019.3.gem
gem "rbpdf", "~> 1.19.6" https://rubygems.org/downloads/rbpdf-1.20.1.gem
gem "net-ldap", "~> 0.16.0" https://rubygems.org/downloads/net-ldap-0.16.1.gem
gem "ruby-openid", "~> 2.9.2", :require => "openid" https://rubygems.org/downloads/ruby-openid-2.9.2.gem
gem "rack-openid" https://rubygems.org/downloads/rack-openid-1.4.2.gem
gem "rmagick", "~> 2.16.0" https://rubygems.org/downloads/rmagick-2.16.0.gem
gem "redcarpet", "~> 3.4.0" https://rubygems.org/downloads/redcarpet-3.4.0.gem
gem "pg", "~> 1.1.4", :platforms => [:mri, :mingw, :x64_mingw] https://rubygems.org/downloads/pg-1.1.4.gem
gem 'mini_portile2' (~> 2.4.0)  https://rubygems.org/downloads/mini_portile2-2.4.0.gem
gem 'tzinfo' (>= 1.0.0) https://rubygems.org/downloads/tzinfo-2.0.0.gem
gem 'concurrent-ruby' (~> 1.0) https://rubygems.org/downloads/concurrent-ruby-1.0.5.gem
gem 'sprockets-rails' (>= 2.0.0) https://rubygems.org/downloads/sprockets-rails-3.2.1.gem
#gem 'railties' (= 6.0.1) https://rubygems.org/downloads/railties-6.0.1.gem
#gem 'actiontext' (= 6.0.1) https://rubygems.org/downloads/actiontext-6.0.1.gem
#gem 'actionmailbox' (= 6.0.1) https://rubygems.org/downloads/actionmailbox-6.0.1.gem
#gem 'activestorage' (= 6.0.1) https://rubygems.org/downloads/activestorage-6.0.1.gem
#gem 'actioncable' (= 6.0.1) https://rubygems.org/downloads/actioncable-6.0.1.gem
#gem 'activejob' (= 6.0.1) https://rubygems.org/downloads/activejob-6.0.1.gem
#gem 'actionmailer' (= 6.0.1) https://rubygems.org/downloads/actionmailer-6.0.1.gem
#gem 'activerecord' (= 6.0.1) https://rubygems.org/downloads/activerecord-6.0.1.gem
#gem 'activemodel' (= 6.0.1) https://rubygems.org/downloads/activemodel-6.0.1.gem
#gem 'actionview' (= 6.0.1) https://rubygems.org/downloads/actionview-6.0.1.gem
#gem 'actionpack' (= 6.0.1) https://rubygems.org/downloads/actionpack-6.0.1.gem
#gem 'activesupport' (= 6.0.1) https://rubygems.org/downloads/activesupport-6.0.1.gem
gem 'method_source' (>= 0) https://rubygems.org/downloads/method_source-0.9.2.gem
gem 'thor' (>= 0.20.3, < 2.0) https://rubygems.org/downloads/thor-0.20.3.gem
gem 'marcel' (~> 0.3.1) https://rubygems.org/downloads/marcel-0.3.1.gem
gem 'websocket-driver' (>= 0.6.1) https://rubygems.org/downloads/websocket-driver-0.7.1.gem
gem 'nio4r' (~> 2.0) https://rubygems.org/downloads/nio4r-2.0.0.gem
gem 'globalid' (>= 0.3.6) https://rubygems.org/downloads/globalid-0.4.2.gem
gem 'rails-dom-testing' (~> 2.0) https://rubygems.org/downloads/rails-dom-testing-2.0.3.gem
gem 'rails-html-sanitizer' (~> 1.1, >= 1.2.0) https://rubygems.org/downloads/rails-html-sanitizer-1.3.0.gem
gem 'erubi' (~> 1.4) https://rubygems.org/downloads/erubi-1.4.0.gem
gem 'builder' (~> 3.1) https://rubygems.org/downloads/builder-3.1.4.gem
gem 'rack-test' (>= 0.6.3) https://rubygems.org/downloads/rack-test-1.1.0.gem
gem 'zeitwerk' (~> 2.2) https://rubygems.org/downloads/zeitwerk-2.2.1.gem
gem 'tzinfo' (~> 1.1) https://rubygems.org/downloads/tzinfo-1.1.0.gem
gem 'websocket-extensions' (>= 0.1.0) https://rubygems.org/downloads/websocket-extensions-0.1.4.gem
gem 'loofah' (~> 2.3) https://rubygems.org/downloads/loofah-2.3.1.gem
gem 'crass' (~> 1.0.2) https://rubygems.org/downloads/crass-1.0.5.gem
gem 'thread_safe' (~> 0.1) https://rubygems.org/downloads/thread_safe-0.1.3.gem
gem 'atomic' (>= 0) https://rubygems.org/downloads/atomic-1.1.101.gem
gem 'railties' (= 5.2.3) https://rubygems.org/downloads/railties-5.2.3.gem
gem 'actiontext' (= 5.2.3) https://rubygems.org/downloads/actiontext-5.2.3.gem
gem 'actionmailbox' (= 5.2.3) https://rubygems.org/downloads/actionmailbox-5.2.3.gem
gem 'activestorage' (= 5.2.3) https://rubygems.org/downloads/activestorage-5.2.3.gem
gem 'actioncable' (= 5.2.3) https://rubygems.org/downloads/actioncable-5.2.3.gem
gem 'activejob' (= 5.2.3) https://rubygems.org/downloads/activejob-5.2.3.gem
gem 'actionmailer' (= 5.2.3) https://rubygems.org/downloads/actionmailer-5.2.3.gem
gem 'activerecord' (= 5.2.3) https://rubygems.org/downloads/activerecord-5.2.3.gem
gem 'activemodel' (= 5.2.3) https://rubygems.org/downloads/activemodel-5.2.3.gem
gem 'actionview' (= 5.2.3) https://rubygems.org/downloads/actionview-5.2.3.gem
gem 'actionpack' (= 5.2.3) https://rubygems.org/downloads/actionpack-5.2.3.gem
gem 'activesupport' (= 5.2.3) https://rubygems.org/downloads/activesupport-5.2.3.gem
gem 'arel' (>= 9.0) https://rubygems.org/downloads/arel-9.0.0.gem
gem 'minitest' (~> 5.1) https://rubygems.org/downloads/minitest-5.1.0.gem
gem 'roadie' (~> 3.1) https://rubygems.org/downloads/roadie-3.1.1.gem
gem 'css_parser' (~> 1.3.4) https://rubygems.org/downloads/css_parser-1.3.4.gem
gem 'nokogiri' (>= 1.5.0, < 1.7.0) https://rubygems.org/downloads/nokogiri-1.6.8.1.gem
gem 'addressable' (>= 0) https://rubygems.org/downloads/addressable-2.7.0.gem
gem 'public_suffix' (>= 2.0.2, < 5.0) https://rubygems.org/downloads/public_suffix-4.0.1.gem
gem 'mini_portile2' (~> 2.1.0) https://rubygems.org/downloads/mini_portile2-2.1.0.gem
gem 'sprockets' (~> 3.7.2 ) https://rubygems.org/downloads/sprockets-3.7.2.gem
gem 'rbpdf (~> 1.19.6)' https://rubygems.org/downloads/rbpdf-1.19.8.gem
gem 'rbpdf-font' (~> 1.19.0) https://rubygems.org/downloads/rbpdf-font-1.19.1.gem
gem 'htmlentities' (>= 0) https://rubygems.org/downloads/htmlentities-4.3.4.gem
gem 'mysql2 (~> 0.5.0)' https://rubygems.org/downloads/mysql2-0.5.2.gem
gem 'yard' https://rubygems.org/downloads/yard-0.9.20.gem
gem 'mocha' https://rubygems.org/downloads/mocha-1.9.0.gem
gem 'metaclass' (~> 0.0.1) https://rubygems.org/downloads/metaclass-0.0.4.gem
gem 'simplecov (~> 0.14.1)' https://rubygems.org/downloads/simplecov-0.14.1.gem
gem 'docile' (~> 1.1.0) https://rubygems.org/downloads/docile-1.1.5.gem
gem 'simplecov-html' (~> 0.10.0) https://rubygems.org/downloads/simplecov-html-0.10.2.gem
gem 'puma (~> 3.7)' https://rubygems.org/downloads/puma-3.7.1.gem
gem 'capybara (~> 2.13)' https://rubygems.org/downloads/capybara-2.13.0.gem
gem 'xpath' (~> 2.0) https://rubygems.org/downloads/xpath-2.0.0.gem
gem 'mime-types' (>= 1.16) https://rubygems.org/downloads/mime-types-3.3.gem
gem 'mime-types-data' (~> 3.2015) https://rubygems.org/downloads/mime-types-data-3.2015.1120.gem
gem 'selenium-webdriver' https://rubygems.org/downloads/selenium-webdriver-3.142.6.gem
gem 'rubyzip' (>= 1.2.2) https://rubygems.org/downloads/rubyzip-1.3.0.gem
gem 'childprocess' (>= 0.5, < 4.0) https://rubygems.org/downloads/childprocess-3.0.0.gem
gem 'rake (>= 0.8.7)' https://rubygems.org/downloads/rake-13.0.1.gem

Положить их в каталог /opt/redmine/redmine-4.0.5/vendor/cache

И установить bundler и passenger локально

$ cd /opt/redmine/redmine-4.0.5/vendor/cache
$ sudo gem install bundler --local
$ sudo gem install passenger --local

Установка NGINX + Passenger

Скачиваем NGINX и устанавливаем его с поддержкой Passenger

$ cd /tmp
$ curl -L https://nginx.org/download/nginx-1.17.5.tar.gz -o /tmp/nginx-1.17.5.tar.gz
$ tar zxf /tmp/nginx-1.17.5.tar.gz
$ sudo passenger-install-nginx-module --prefix=/opt/nginx --nginx-source-dir=/tmp/nginx-1.17.5 --languages ruby

Enter your choice (1 or 2) or press Ctrl-C to abort: 2 (собираем из исходников)
Please specify the directory: /tmp/nginx-1.17.5
Please specify a prefix directory [/opt/nginx]:

--------------------------------------------

Nginx with Passenger support was successfully installed.

The Nginx configuration file (/opt/nginx/conf/nginx.conf)
must contain the correct configuration options in order for Phusion Passenger
to function correctly.

This installer has already modified the configuration file for you! The
following configuration snippet was inserted:

  http {
      ...
      passenger_root /usr/local/lib/ruby/gems/2.6.0/gems/passenger-6.0.4;
      passenger_ruby /usr/local/bin/ruby;
      ...
  }

After you start Nginx, you are ready to deploy any number of Ruby on Rails
applications on Nginx.

Press ENTER to continue.

-------------------------------------------------

Для удобства создаём симлинк

$ sudo ln -s /opt/nginx/conf/ /etc/nginx

Редактируем файл конфигурации Nginx:

$ sudo nano /opt/nginx/conf/nginx.conf
user redmine;
worker_processes  auto;
...

В блок http добавляем следующий текст (в самом низу)

...
include sites-enabled/*.conf;
server_names_hash_bucket_size 64;
}

Далее настраиваем доступ к хосту Redmine

$ sudo mkdir /opt/nginx/conf/{sites-available,sites-enabled}
$ sudo nano /opt/nginx/conf/sites-available/redmine.conf

Пример конфигурационного файла для nginx

$ cat redmine.conf
server {
    listen 80 default_server;
    server_name redmine.example.com;

    gzip on;
    gzip_comp_level 7;
    gzip_types application/x-javascript application/javascript text/css;

    set $root_path /opt/redmine/redmine-latest/public;
    root $root_path;

    passenger_enabled on;
    passenger_document_root $root_path;
    client_max_body_size      100m; # Max attachemnt size
    client_body_buffer_size 4M;

    # redirect server error pages to the static page /50x.html
    error_page   500 502 503 504  /50x.html;

    location = /50x.html {
        root   html;
    }
		
    location ~* \.(jpg|gif|png|js|css|ico)$ {
        root $root_path;
        expires 7d;
    }

}

Делаем симлинк

$ cd /opt/nginx/conf/sites-available
$ sudo ln -s /opt/nginx/conf/sites-available/redmine.conf /opt/nginx/conf/sites-enabled/redmine.conf

Проверяем nginx

$ sudo /opt/nginx/sbin/nginx -t

Создаем файл для запуска nginx в качестве сервиса

$ sudo nano /lib/systemd/system/nginx.service
[Unit]
Description=The NGINX HTTP and reverse proxy server
After=syslog.target network.target remote-fs.target nss-lookup.target
 
[Service]
Type=forking
PIDFile=/opt/nginx/logs/nginx.pid
ExecStartPre=/opt/nginx/sbin/nginx -t
ExecStart=/opt/nginx/sbin/nginx
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true
 
[Install]
WantedBy=multi-user.target

Перечитываем конфигурации systemd

$ sudo systemctl daemon-reload

Запускаем Nginx и добавляем его в автозагрузку

$ sudo systemctl start nginx
$ sudo systemctl enable nginx

Устанавливаем Redmine. Продолжение

Переходим в каталог

$ cd /opt/redmine/redmine-4.0.5/

Прописываем настройки для PostgreSQL

$ bundle config build.pg --with-pg-config=/usr/pgsql-10/bin/pg_config

Устанавливаем необходимые gems (локальная установка)

$ bundle install --without development test mysql2 sqlite --local

Либо, если у сервера есть выход в интернет, устанавливаем необходимые gems:

$ bundle install --without development test mysql2 sqlite

Запускаем генерацию токена:

$ bundle exec rake generate_secret_token

Создаем структуру БД

$ RAILS_ENV=production bundle exec rake db:migrate

Загружаем в базу дефолтные данные

$ RAILS_ENV=production bundle exec rake redmine:load_default_data
  Select language: ru

Установка приложения Redmine завершена. Меняем владельца и прав доступа к каталогам и файлам

$ mkdir -p tmp tmp/pdf public/plugin_assets
$ chown -R redmine:redmine files log tmp public/plugin_assets
$ chmod -R 755 files log tmp public/plugin_assets

Перезагружаем nginx

$ sudo systemctl restart nginx

Осталось поменять пароль админа, для этого открываем браузер, переходим на соответствующую страницу и меняем пароль

Настройка LDAP AD / FreeIPA

Настройка параметров LDAP производится через web-интерфейс

Пример параметров для подключения LDAP Active Directory

Имя: example.com
Компьютер: 192.168.0.2
Порт: 389 LDAP
Учётная запись: user@example.com
Пароль: •••••••••••••••
BaseDN: dc=example,dc=com
Создание пользователя на лету
Атрибут Login: sAMAccountName
Имя: givenName
Фамилия: sn
email: mail

Пример параметров для подключения LDAP FreeIPA

Имя: FreeIPA
Компьютер: freeipa.example.com
Порт: 636 LDAPS без проверки сертификата
Учётная запись
Пароль
BaseDN: cn=accounts,dc=example,dc=com
Создание пользователя на лету
Атрибут Login: uid
Имя: givenName
Фамилия: sn
email: mail

Установка Redmine 4.0.5 NGINX PostgreSQL в Centos 7

Leave a comment Posted on 02.11.202102.11.2021 by 891rpm Без рубрики

Установка Redmine 2.4.0

Redmine — это информационная система управления проектами с веб интерфейсом (онлайн), включающая в себя полный набор средств для совместной работы над проектами. Система позволяет вести одновременно несколько проектов, отслеживать их состояния, управлять шагами проекта, задачами, приоритетами, гибко назначать роли участникам. Распространяется по лицензии GNU. (Т.е. продукт бесплатен даже для коммерческого использования и не накладывает никаких ограничений на количество пользователей). Это весомый аргумент для многих клиентов, так как лицензирование является существенной статьей бюджета для многих компаний. Наша компания занимается полным циклом внедрения систем управления проектами и №1 среди них, конечно же Redmine. Мы сами пользуемся этой системой управления проектами уже более 3х лет и она показала себя только с хороших сторон — как производительная, удобная в использовании и настройке и очень дружелюбная к пользователям.

Какие преимущества дает система Redmine и какие задачи она решает?

Организует единый центр ведения проектов, программ и портфелей проектов в компании с гибкими настройками ролей участников – один и тот же сотрудник может играть разные роли в разных проектах. Обеспечивается единый стандарт ведения проектов в организации.
Позволяет вовлечь участников проекта в процесс, обеспечить визуальное представление задач, сроков, вех проекта. Все знают, что делать дальше и видят цель.
Гибкая отчетность по проектам: кто, что и когда делал, делает и будет делать. Видимость загруженности ресурсов, контроль сроков, история задач. Автоматическое построение диаграммы Ганта и отображения задач на календарном плане.
Простота доступа к информации из любой точки, в том числе для географически удаленных сотрудников и подразделений. Возможность доступа заинтересованных лиц, спонсоров и других участников проекта, явно не связанных с выполнением задач, к информации и отчетности в режиме просмотра.
Система решает задачу социального взаимодействия в проектах, предоставляя встроенные проектные форумы (средства для обсуждений), доски новостей, базы знаний и возможность комментировать и обсуждать задачи.
Возможность настройки продукта на любую предметную область бизнеса, путем введения новых справочников, дополнительных полей к задачам, схем обработки последовательности задач.
Инструмент не только для проектного менеджера, но и всех участников проектной команды, предоставляет доступ к проекту всем всегда и везде, в том числе с мобильных устройств.

Redmine — открытое серверное веб-приложение для управления проектами и задачами (в том числе для отслеживания ошибок). Redmine написан на Ruby и представляет собой приложение на основе широко известного веб-фреймворка Ruby on Rails. Распространяется согласно GNU General Public License.

Устанавливаем необходимые библиотеки

[root@localhost]# yum install make gcc gcc-c++ zlib-devel curl-devel openssl-devel httpd-devel apr-devel apr-util-devel mysql-devel
[root@localhost]# yum install zlib zlib-devel openssl-devel sqlite-devel gcc-c++ glibc-headers libyaml-devel readline readline-devel zlib-devel libffi-devel

Скачиваем исходники Ruby

[root@localhost]# cd ~
[root@localhost]# wget http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.5.tar.gz

Распаковываем

[root@localhost]# tar zxvf ruby-2.1.5.tar.gz

Компилируем и устанавливаем

[root@localhost]# cd ruby-2.1.5
[root@localhost]# ./configure
[root@localhost]# make
[root@localhost]# make install

Смотрим версию

[root@localhost]# ruby -v

Устанавливаем passenger:

[root@localhost]# gem install passenger
[root@localhost]# passenger-install-apache2-module

Создаем конфигурационный файл

[root@localhost]# nano /etc/httpd/conf.d/passenger.conf

LoadModule passenger_module /usr/local/lib/ruby/gems/2.1.0/gems/passenger-5.0.4/buildout/apache2/mod_passenger.so
<IfModule mod_passenger.c>
   PassengerRoot /usr/local/lib/ruby/gems/2.1.0/gems/passenger-5.0.4
   PassengerDefaultRuby /usr/local/bin/ruby
</IfModule>

Перезапускаем Apache

[root@localhost]# service httpd restart

Настройки хоста для Apache:

<VirtualHost *:80>
ServerName www.yourhost.com
# !!! Be sure to point DocumentRoot to 'public'!
DocumentRoot /somewhere/public 
<Directory /somewhere/public>
# This relaxes Apache security settings.
AllowOverride all
# MultiViews must be turned off.
Options -MultiViews
# Uncomment this if you're on Apache >= 2.4:
#Require all granted
</Directory>
</VirtualHost>

Качаем Redmine

[root@localhost]# cd ~
[root@localhost]# wget http://www.redmine.org/releases/redmine-2.4.0.tar.gz

Распаковываем

[root@localhost]# tar zxvf redmine-2.4.0.tar.gz

Переносим распакованные файлы в /var/www/html/redmine

[root@localhost]# mv redmine-2.4.0/* /var/www/redmine

Ставим

[root@localhost]# gem install bundle

Меняем владельца директории

[root@localhost]# chown -R apache:apache /var/www/html/redmine

Доустанавливаем библиотеки

[root@localhost]# yum install ImageMagick-devel
[root@localhost]# gem install rmagick -v '2.13.2'

Устанавливаем redmine

[root@localhost]# cd /var/www/redmine
[root@localhost]# bundle install --without postgresql sqlite test development

Настройка подключения к базе

[root@localhost]# mysql -u root -p
mysql> create database redmine character set utf8;
mysqk> grant all privileges on redmine.* to 'redmine'@'localhost' identified by 'redmine';
mysql> flush privileges;
mysql> quit;

Конфигурируем redmine для подключения к базе

[root@localhost]# cd /var/www/html/redmine/config
[root@localhost]# cp database.yml.example database.yml

Открываем database.yml и прописываем логин/пароль от базы

[root@localhost]# nano database.yml

переходим в каталог и доустанавливаем

[root@localhost]# cd /var/www/html/redmine
[root@localhost]# bundle install
[root@localhost]# rake generate_secret_token

Первичное заполнение базы

[root@localhost]# rake db:migrate RAILS_ENV="production"
[root@localhost]# rake redmine:load_default_data RAILS_ENV="production"

Установка плагинов

[root@localhost]# cd /var/www/html/redmine/plugins

# redmine_multiprojects_issue

[root@localhost]# wget https://github.com/nanego/redmine_multiprojects_issue/archive/master.zip
[root@localhost]# unzip masters.zip
[root@localhost]# bundle install
[root@localhost]# rake redmine:plugins:migrate RAILS_ENV=production
[root@localhost]# rm master.zip

# redmine_base_select2

[root@localhost]# wget https://github.com/jbbarth/redmine_base_select2/archive/master.zip
[root@localhost]# unzip masters.zip
[root@localhost]# bundle install
[root@localhost]# rake redmine:plugins:migrate RAILS_ENV=production
[root@localhost]# rm master.zip

# redmine_base_deface

[root@localhost]# wget https://github.com/jbbarth/redmine_base_deface/archive/master.zip
[root@localhost]# unzip masters.zip
[root@localhost]# bundle install
[root@localhost]# rake redmine:plugins:migrate RAILS_ENV=production
[root@localhost]# rm master.zip

Перезапускаем Apache

[root@localhost]# service httpd restart

Установка Redmine 2.4.0

Leave a comment Posted on 01.01.2021 by 891rpm Без рубрики

Ejabberd XMPP server

В этой статье будет описана поэтапная установка и настройка jabber-сервера ejabberd в Debian-совместимых системах (Ubuntu, Linux Mint и т.п.).

В первую очередь, перед установкой самого сервера, рекомендуем внести DNS-записи для будущего сервера, новые записи, как правило, активизируются не сразу. Может пройти до 72 часов до того, как они вступят в силу. Допустим, вы устанавливаете xmpp-сервер на домен myxmpp.ru, для этого у регистратора (или у того, чьи DNS-сервера вы используете) надо внести две SRV-записи:

_xmpp-server._tcp.myxmpp.ru. 86400 IN SRV 5 0 5269 myxmpp.ru.
_xmpp-client._tcp.myxmpp.ru. 86400 IN SRV 5 0 5222 myxmpp.ru.

Здесь вы можете подробнее прочитать о DNS SRV-записях и их проверке.

Они нужны для успешного s2s (server to server) сообщения, если вы не планируете связываться с другими xmpp-серверами, а делаете, например, внутренний корпоративный сервер – то вносить эти DNS-записи нет необходимости.

Для установки свежей версии ejabberd рекомендуем использовать apt-репозиторий apt.jabber.at. Добавляется он так, под рутом введите:

apt-get install apt-transport-https
echo "deb https://apt.jabber.at <dist> ejabberd" \
> /etc/apt/sources.list.d/jabber.at.list
wget -qO- https://apt.jabber.at/gpg-key | apt-key add -
apt-get update

Где <dist> замените на название вашего дистрибутива, например на jessie. Список поддерживаемых репозиторием дистрибутивов можно посмотреть здесь.

После этого для установки свежей версии xmpp-сервера достаточно сделать:

apt-get install ejabberd

Далее, рекомендуем использовать доверенный сертификат, а не самоподписанный, для того, чтобы im-клиенты не задавали лишних вопросов при добавлении вашего сервера. Если вы используете сертификаты от letsencrypt, то для создания файла для ejabberd вам понадобится помимо сертификатов самого домена ещё и Intermediate Certificates Authority X3. Загрузите его:

wget https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt

И если у вас уже установлены сертификаты letsencrypt для вашего домена, то создайте файл:

cat /etc/letsencrypt/live/myxmpp.ru/privkey.pem /etc/letsencrypt/live/myxmpp.ru/fullchain.pem ~/lets-encrypt-x3-cross-signed.pem.txt >> /etc/ejabberd/xmpp.pem

(где “~/lets-encrypt-x3-cross-signed.pem.txt” – путь к загруженному ранее intermediate-сертификату)

Теперь перейдём к редактированию конфигурационного файла /etc/ejabberd/ejabberd.yml (может быть ejabberd.cfg в некоторых версиях). Ниже мы отметим только критичные для минимальной установки строки, которые необходимо отредактировать:

hosts:
  - "myxmpp.ru"

listen:
  -
    port: 5222
    certfile: "/etc/ejabberd/xmpp.pem"
  -
    port: 5280
    certfile: "/etc/ejabberd/xmpp.pem"
    s2s_certfile: "/etc/ejabberd/xmpp.pem"
    s2s_use_starttls: optional

acl:
  admin:
    user:
      - "admin@myxmpp.ru"

Отмеченное цветом это – то, что должно быть обязательно расскоментированно и необходимо изменить в соответствии, с вашим доменом и расположением файла сертификата.

Значение s2s_use_starttls: true долнжо быть именно optional для того, чтобы работала s2s связь c gmail.com (google talk). Иначе, при попытке связаться с контактами google (хотя с другими XMPP-серверами соединение будет проходить успешно) будет ошибка:

Closing s2s connection: myxmpp.ru -> gmail.com (invalid dialback key)

Наконец, создайте пользователя, которому выделены права администратора:

ejabberdctl register admin myxmpp.ru 123456

Где “123456” – пароль.

И перезагрузите ejabberd:

/etc/init.d/ejabberd restart

Всё, после этого xmpp-сервер ejabberd должен работать. Можете добавлять пользовотеля admin@myxmpp.ru в xmpp-клиент, создавать по-аналогии других пользователей, изменять конфигурацию, а также использовать администраторский веб-интерфейс по адресу:

https://myxmpp.ru:5280/admin

– с логином и паролем указанными выше.

Если после указанных настроек s2s не работает, не происходит авторизации и в логе подобные ошибки:

@ejabberd_s2s_out:srv_lookup:1215 The DNS servers timed out on request for "somedomain.com" IN SRV. You should check your DNS configuration.
@ejabberd_s2s_out:open_socket:271 s2s connection: myxmpp.ru -> somedomain.com (remote server not found)
@ejabberd_s2s_out:handle_info:990 Reconnect delay expired: Will now retry to connect to somedomain.com when needed.

– проверьте свой resolv.conf подобно тому, как это описано здесь.

Leave a comment Posted on 17.06.2020 by 891rpm Без рубрики

How to update a Jenkins job posting config.xml

Recently I wanted to update a few jobs (not defined with a DSL) in Cloudbees, adding to each of them a few properties.

Well, I had some trouble making it work, here are my notes (I used Jenkins 1.651.2.1 but chances are it should work with older and more recent versions, such as jenkins 2)

Contents

1 No security / no auth
2 Simple security : using username and password
3 Simple security : with CSRF enabled (crumb)
3.1 Get the crumb value :
3.2 Use the crumb value
4 Security, based on cookies (no user / password)
5 Links

No security / no auth

This is the easy part : retrieve and re post the configuration

$ curl http://localhost:8080/jenkins/job/pof/config.xml -o config.xml
$ curl -X POST http://localhost:8080/jenkins/job/pof/config.xml --data-binary @config.xml

Simple security : using username and password

I assume now that your Jenkins set up has security set up (http://localhost:8080/jenkins/configureSecurity/ –> enable security)

It means we now need to authenticate both our requests :

curl -X GET http://anthony:anthony@localhost:8080/jenkins/job/pof/config.xml -o config.xml
curl -X POST http://anthony:anthony@localhost:8080/jenkins/job/pof/config.xml --data-binary "@config.xml"

Simple security : with CSRF enabled (crumb)

You would also want to protect your jenkins instance against CSRF attacks (http://localhost:8080/jenkins/configureSecurity/ –> enable csrf crumb)

Now, it also means your requests need to send a crumb value, either as a parameter or via a header

If you don’t :

curl -X POST http://anthony:anthony@localhost:8080/jenkins/job/pof/config.xml --data-binary "@config.xml"

You’ll get such errors :

<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /jenkins/job/pof/config.xml. Reason:
<pre> No valid crumb was included in the request</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/>

or even :

<body><h1>HTTP Status 500 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The server encountered an internal error () that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>java.io.IOException: Failed to persist config.xml
hudson.model.AbstractItem.updateByXml(AbstractItem.java:677)
hudson.model.AbstractItem.doConfigDotXml(AbstractItem.java:617)
…..
</pre></p><p><b>root cause</b> <pre>javax.xml.transform.TransformerException: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Premature end of file.
com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:755)
com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:357)
jenkins.util.xml.XMLUtils._transform(XMLUtils.java:96)
jenkins.util.xml.XMLUtils.safeTransform(XMLUtils.java:63)
hudson.model.AbstractItem.updateByXml(AbstractItem.java:674)
hudson.model.AbstractItem.doConfigDotXml(AbstractItem.java:617)

Get the crumb value :

You can either the crumb value using a configure job page :

curl http://anthony:anthony@localhost:8080/jenkins/job/pof/configure | sed -n 's/.*\.crumb", "\(.*\)").*/\1/p' > crumb.txt

But there’s also a service dedicated to that :

curl http://anthony:anthony@localhost:8080/jenkins/crumbIssuer/api/xml | sed -n 's/.*<crumb>\(.*\)<\/crumb>.*/\1/p' > crumb.txt

Use the crumb value

curl -X POST http://anthony:anthony@localhost:8080/jenkins/job/pof/config.xml --data-binary "@config.xml" -data ".crumb=6bbabc426436b72ec35e5ad4a4344687"

Oups, that did not work

Caused by: java.lang.IllegalStateException: STREAMED
at org.eclipse.jetty.server.Request.getReader(Request.java:803)
at javax.servlet.ServletRequestWrapper.getReader(ServletRequestWrapper.java:256)
at hudson.model.AbstractItem.doConfigDotXml(AbstractItem.java:610)

I suggest you send the crumb using the headers :

curl -v -X POST http://anthony:anthony@localhost:8080/jenkins/job/pof/config.xml --data-binary "@config.xml" -H ".crumb: 6bbabc426436b72ec35e5ad4a4344687"

Security, based on cookies (no user / password)

In some installs, such as cloubees, you can’t pass username and password in your requests; I suggest you use the cookies instead.

To retrieve them, inspect the cookies sent by your authenticated browser, for example in chrome :

And then paste this url in your shell :

curl 'http://localhost:8080/jenkins/job/pof/config.xml' -H 'Pragma: no-cache' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8,fr;q=0.6' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Referer: http://localhost:8080/jenkins/login?from=%2Fjenkins%2Fjob%2Fpof%2Fconfig.xml' -H 'Cookie: screenResolution=1440x900; JSESSIONID=XXXXX; JSESSIONID.XX=XXXX; screenResolution=1440x900' -H 'Connection: keep-alive' -H 'Cache-Control: no-cache' --compressed

Of course, you’ll still need to get the crumb value :

curl 'http://localhost:8080/jenkins/crumbIssuer/api/xml' -H 'Pragma: no-cache' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8,fr;q=0.6' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Referer: http://localhost:8080/jenkins/login?from=%2Fjenkins%2Fjob%2Fpof%2Fconfig.xml' -H 'Cookie: screenResolution=1440x900; JSESSIONID=XXXXX; JSESSIONID.XX=XXXXX; screenResolution=1440x900' -H 'Connection: keep-alive' -H 'Cache-Control: no-cache' --compressed | sed -n 's/.*<crumb>\(.*\)<\/crumb>.*/\1/p' > crumb.txt

And now you’re ready to post your updated config.xml :

curl -X POST 'http://localhost:8080/jenkins/job/pof/config.xml' -H 'Pragma: no-cache' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8,fr;q=0.6' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Referer: http://localhost:8080/jenkins/login?from=%2Fjenkins%2Fjob%2Fpof%2Fconfig.xml' -H 'Cookie: screenResolution=1440x900; JSESSIONID=XXXX; JSESSIONID.XX=XXXX; screenResolution=1440x900' -H 'Connection: keep-alive' -H 'Cache-Control: no-cache' --compressed --data-binary "@config.xml" -H ".crumb: 6bbabc426436b72ec35e5ad4a4344687"

Links

https://benkiew.wordpress.com/2012/01/12/automating-hudsonjenkins-via-rest-and-curl-a-very-small-cookbook/
https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API
https://blog.dahanne.net/2016/05/17/how-to-update-a-jenkins-job-posting-config-xml/

Leave a comment Posted on 24.01.202024.01.2020 by 891rpm Без рубрики

14 Best Russian Language Datasets for Machine Learning

Despite recent advances in machine learning, developing Russian natural language processing (NLP) systems remains a big challenge for researchers. Russian, like other Slavic languages, is a morphologically rich language with a free word order and large degree of inflection. These linguistic factors make it difficult to collect enough relevant multilingual data to train machine learning models.

To help, we at Lionbridge AI have put together an exhaustive list of the best Russian datasets available on the web, covering everything from social media data to natural speech.

Russian Text Datasets

Russian Common Crawl Data: Over 500 terabytes of raw Russian text data crawled from the web.

OpenCorpora Russian: A tagged corpus of 1.5 million Russian words encoded in UTF-8.

Uppsala Russian Corpus: A million word corpus designed to be as representative and varied as possible. The texts cover 25 different subject areas, such as economics, law, and medicine.

Taiga Corpus: 6.5 million tokens in Russian available upon request.

Classification of Handwritten Letters: Over 10,000 images of handwritten Russian characters useful for training text classification and image generation systems.

StimulStat: A lexical database for Russian which allows selecting words and word forms based on different parameters for a list of words or word forms.

NLP datasets for Russian language: A repository that includes a variety of NLP datasets and resources in Russian, including conversation dialogues, word lists, question-answer data and more.

Russian Social Media Datasets

Russian Twitter Corpus: A corpus of 17.6 million tweets in Russian suitable for training language model for social media, content moderation, sentiment analysis and more.

Russian Troll Tweets: A dataset containing over 200,000 malicious-account tweets captured by NBC.

SentiRuEval trainset: Over 20,000 Russian tweets tagged with sentiment data.

Russian Audio Datasets

Russian Speech Database: Recorded in 1996-1998, the STC Russian speech database was created to investigate individual speaker variability and validate speaker recognition algorithms.

Russian Single Speaker Speech Dataset: CSS10 is a collection of single speaker speech datasets for 10 languages, including Russian.

M-AILABS Speech Dataset: A large Russian audio dataset, freely usable as training data for speech recognition and speech synthesis.

Russian Open Speech To Text (STT/ASR) Dataset: A dataset containing 4,000+ of diverse, cross domain speech to train speech-to-text models in Russian.

Still can’t find what you’re looking for? Lionbridge has extensive experience creating custom annotated data in Russian for a wide range of machine learning applications. With 20 years of experience and 500,000+ qualified native speakers around the world, multilingual datasets are Lionbridge’s strength.

Alex Nguyen

https://lionbridge.ai/datasets/best-russian-language-datasets-for-machine-learning/

Posted on 01.01.202006.09.2026 by 891rpm Без рубрики

x

Kamikaze

Posted on 01.01.202004.09.2026 by 891rpm Без рубрики

x

Kamikaze

Posted on 01.01.202030.08.2026 by 891rpm Без рубрики

x

SlotMonster Casino: Snelle Winsten & Intense Slot Actie voor Mobiel Spel

Leave a comment Posted on 29.04.2019 by 891rpm Tech articles

A simple filtering syntax tree in Python

Working on various pieces of software those last years, I noticed that there’s always a feature that requires implementing some DSL.

The problem with DSL is that it is never the road that you want to go. I remember how creating my first DSL was fascinating: after using programming languages for years, I was finally designing my own tiny language!

A new language that my users would have to learn and master. Oh, it had nothing new, it was a subset of something, inspired by my years of C, Perl or Python, who knows. And that’s the terrible part about DSL: they are an marvelous tradeoff between the power that they give to users, allowing them to define precisely their needs and the cumbersomeness of learning a language that will be useful in only one specific situation.

In this blog post, I would like to introduce a very unsophisticated way of implementing the syntax tree that could be used as a basis for a DSL. The goal of that syntax tree will be filtering. The problem it will solve is the following: having a piece of data, we want the user to tell us if the data matches their conditions or not.

To give a concrete example: a machine wants to grant the user the ability to filter the beans that it should keep. What the machine passes to the filter is the size of the current grain, and the filter should return either true or false, based on the condition defined by the user: for example, only keep beans that are bigger that are between 1 and 2 centimeters or between 4 and 6 centimeters.

The number of conditions that the users can define could be quite considerable, and we want to provide at least a basic set of predicate operators: equal, greater than and lesser than. We also want the user to be able to combine those, so we’ll add the logical operators or and and.

A set of conditions can be seen as a tree, where leaves are either predicates, and in that case, do not have children, or are logical operators, and have children. For example, the propositional logic formula φ1 ∨ (φ2 ∨ φ3) can be represented with as a tree like this:

Starting with this in mind, it appears that the natural solution is going to be recursive: handle the predicate as terminal, and if the node is a logical operator, recurse over its children.
Since we will be doing Python, we’re going to use Python to evaluate our syntax tree.

The simplest way to write a tree in Python is going to be using dictionaries. A dictionary will represent one node and will have only one key and one value: the key will be the name of the operator (equal, greater than, or, and…) and the value will be the argument of this operator if it is a predicate, or a list of children (as dictionaries) if it is a logical operator.

For example, to filter our bean, we would create a tree such as:

{"or": [
  {"and": [
    {"ge": 1},
    {"le": 2},
  ]},
  {"and": [
    {"ge": 4},
    {"le": 6},
  ]},
]}

The goal here is to walk through the tree and evaluate each of the leaves of the tree and returning the final result: if we passed 5 to this filter, it would return True, and if we passed 10 to this filter, it would return False.

Here’s how we could implement a very depthless filter that only handles predicates (for now):

import operator

class InvalidQuery(Exception):
    pass

class Filter(object):
    binary_operators = {
        "eq": operator.eq,
        "gt": operator.gt,
        "ge": operator.ge,
        "lt": operator.lt,
        "le": operator.le,
    }

    def __init__(self, tree):
        # Parse the tree and store the evaluator
        self._eval = self.build_evaluator(tree)

    def __call__(self, value):
        # Call the evaluator with the value
        return self._eval(value)

    def build_evaluator(self, tree):
        try:
            # Pick the first item of the dictionary.
            # If the dictionary has multiple keys/values
            # the first one (= random) will be picked.
            # The key is the operator name (e.g. "eq")
            # and the value is the argument for it
            operator, nodes = list(tree.items())[0]
        except Exception:
            raise InvalidQuery("Unable to parse tree %s" % tree)
        try:
            # Lookup the operator name
            op = self.binary_operators[operator]
        except KeyError:
            raise InvalidQuery("Unknown operator %s" % operator)
        # Return a function (lambda) that takes
        # the filtered value as argument and returns
        # the result of the predicate evaluation
        return lambda value: op(value, nodes)

You can use this Filter class by passing a predicate such as {"eq": 4}:

>>> f = Filter({"eq": 4})
>>> f(2)
False
>>> f(4)
True

This Filter class works but is quite limited as we did not provide logical operators. Here’s a complete implementation that supports binary operators and and or:

import operator


class InvalidQuery(Exception):
    pass


class Filter(object):
    binary_operators = {
        u"=": operator.eq,
        u"==": operator.eq,
        u"eq": operator.eq,

        u"<": operator.lt,
        u"lt": operator.lt,

        u">": operator.gt,
        u"gt": operator.gt,

        u"<=": operator.le,
        u"≤": operator.le,
        u"le": operator.le,

        u">=": operator.ge,
        u"≥": operator.ge,
        u"ge": operator.ge,

        u"!=": operator.ne,
        u"≠": operator.ne,
        u"ne": operator.ne,
    }

    multiple_operators = {
        u"or": any,
        u"∨": any,
        u"and": all,
        u"∧": all,
    }

    def __init__(self, tree):
        self._eval = self.build_evaluator(tree)

    def __call__(self, value):
        return self._eval(value)

    def build_evaluator(self, tree):
        try:
            operator, nodes = list(tree.items())[0]
        except Exception:
            raise InvalidQuery("Unable to parse tree %s" % tree)
        try:
            op = self.multiple_operators[operator]
        except KeyError:
            try:
                op = self.binary_operators[operator]
            except KeyError:
                raise InvalidQuery("Unknown operator %s" % operator)
            return lambda value: op(value, nodes)
        # Iterate over every item in the list of the value linked
        # to the logical operator, and compile it down to its own
        # evaluator.
        elements = [self.build_evaluator(node) for node in nodes]
        return lambda value: op((e(value) for e in elements))

To support the and and or operators, we leverage the all and any built-in Python functions. They are called with an argument that is a generator that evaluates each one of the sub-evaluator, doing the trick.

Unicode is the new sexy, so I’ve also added Unicode symbols support.

And it is now possible to implement our full example:

>>> f = Filter(
...     {"∨": [
...         {"∧": [
...             {"≥": 1},
...             {"≤": 2},
...         ]},
...         {"∧": [
...             {"≥": 4},
...             {"≤": 6},
...         ]},
...     ]})
>>> f(5)
True
>>> f(8)
False
>>> f(1)
True

As an exercise, you could try to add the not operator, which deserve its own category as it is a unary operator!

In the next blog post, we will see how to improve that filter with more features, and how to implement a domain-specific language on top of it, to make humans happy when writing the filter!

Hole and Henni – *François Charlier*, 2018

In this drawing, the artist represents the deepness of functional programming and how its horse power can help you escape many dark situations.

Julien Danjou

https://julien.danjou.info/simple-filtering-syntax-tree-in-python/